[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:31:55 dist-test-slave-70rm krb5kdc[2063](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:31:55 dist-test-slave-70rm krb5kdc[2063](info): set up 2 sockets
Jun 19 14:31:55 dist-test-slave-70rm krb5kdc[2063](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260619 14:31:56.066628 2056 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Jun 19 14:31:56 dist-test-slave-70rm krb5kdc[2063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:31:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:31:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.2.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab.
Entry for principal kudu/127.2.2.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab.
I20260619 14:31:56.190289 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:40873
--webserver_interface=127.2.2.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:45663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.2.62:40873
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:31:56.304594 2076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:31:56.304970 2076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:31:56.305084 2076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:31:56.310148 2076 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:31:56.310235 2076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:31:56.310272 2076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:31:56.310310 2076 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:31:56.310345 2076 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:31:56.315922 2076 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:45663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.2.62:40873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:40873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.2.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:31:56.317065 2076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:31:56.318233 2076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:31:56.327270 2081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:31:56.327261 2082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:31:56.327267 2084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:31:56.327610 2076 server_base.cc:1034] running on GCE node
I20260619 14:31:56.328143 2076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:31:56.329329 2076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:31:56.330478 2076 hybrid_clock.cc:648] HybridClock initialized: now 1781879516330400 us; error 93 us; skew 500 ppm
Jun 19 14:31:56 dist-test-slave-70rm krb5kdc[2063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879516, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:31:56.338181 2076 init.cc:377] Logged in from keytab as kudu/127.2.2.62@KRBTEST.COM (short username kudu)
I20260619 14:31:56.339138 2076 webserver.cc:460] Webserver started at http://127.2.2.62:33353/ using document root <none> and password file <none>
I20260619 14:31:56.339716 2076 fs_manager.cc:359] Metadata directory not provided
I20260619 14:31:56.339809 2076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:31:56.340036 2076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:31:56.342078 2076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/master-0/data/instance:
uuid: "351c72276fbd4fb692f8d3da58b72666"
format_stamp: "Formatted at 2026-06-19 14:31:56 on dist-test-slave-70rm"
server_key: "8342504122b4ca4f7d260e064e4c6814"
server_key_iv: "d8f247bfe9e7d4295b33dcac0f29238a"
server_key_version: "encryptionkey@0"
I20260619 14:31:56.342722 2076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/master-0/wal/instance:
uuid: "351c72276fbd4fb692f8d3da58b72666"
format_stamp: "Formatted at 2026-06-19 14:31:56 on dist-test-slave-70rm"
server_key: "8342504122b4ca4f7d260e064e4c6814"
server_key_iv: "d8f247bfe9e7d4295b33dcac0f29238a"
server_key_version: "encryptionkey@0"
I20260619 14:31:56.346478 2076 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:31:56.349031 2091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:31:56.350390 2076 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:31:56.350637 2076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/master-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/master-0/wal
uuid: "351c72276fbd4fb692f8d3da58b72666"
format_stamp: "Formatted at 2026-06-19 14:31:56 on dist-test-slave-70rm"
server_key: "8342504122b4ca4f7d260e064e4c6814"
server_key_iv: "d8f247bfe9e7d4295b33dcac0f29238a"
server_key_version: "encryptionkey@0"
I20260619 14:31:56.350775 2076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:31:56.372169 2076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:31:56.375180 2076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:31:56.375407 2076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:31:56.382588 2076 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.62:40873
I20260619 14:31:56.382611 2143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.62:40873 every 8 connection(s)
I20260619 14:31:56.383587 2076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
I20260619 14:31:56.386459 2144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:31:56.387460 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 2076
I20260619 14:31:56.387630 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/master-0/wal/instance
I20260619 14:31:56.389225 2056 external_mini_cluster.cc:1428] Setting key a9687a6b089ee065570c242c6466423e
I20260619 14:31:56.393463 2144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 351c72276fbd4fb692f8d3da58b72666: Bootstrap starting.
I20260619 14:31:56.396301 2144 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 351c72276fbd4fb692f8d3da58b72666: Neither blocks nor log segments found. Creating new log.
I20260619 14:31:56.397241 2144 log.cc:824] T 00000000000000000000000000000000 P 351c72276fbd4fb692f8d3da58b72666: Log is configured to *not* fsync() on all Append() calls
I20260619 14:31:56.399834 2144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 351c72276fbd4fb692f8d3da58b72666: No bootstrap required, opened a new log
I20260619 14:31:56.404774 2144 raft_consensus.cc:348] T 00000000000000000000000000000000 P 351c72276fbd4fb692f8d3da58b72666 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "351c72276fbd4fb692f8d3da58b72666" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 40873 } }
I20260619 14:31:56.405030 2144 raft_consensus.cc:374] T 00000000000000000000000000000000 P 351c72276fbd4fb692f8d3da58b72666 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:31:56.405066 2144 raft_consensus.cc:729] T 00000000000000000000000000000000 P 351c72276fbd4fb692f8d3da58b72666 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 351c72276fbd4fb692f8d3da58b72666, State: Initialized, Role: FOLLOWER
I20260619 14:31:56.405615 2144 consensus_queue.cc:260] T 00000000000000000000000000000000 P 351c72276fbd4fb692f8d3da58b72666 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "351c72276fbd4fb692f8d3da58b72666" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 40873 } }
I20260619 14:31:56.405783 2144 raft_consensus.cc:388] T 00000000000000000000000000000000 P 351c72276fbd4fb692f8d3da58b72666 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:31:56.405853 2144 raft_consensus.cc:482] T 00000000000000000000000000000000 P 351c72276fbd4fb692f8d3da58b72666 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:31:56.405956 2144 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 351c72276fbd4fb692f8d3da58b72666 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:31:56.407267 2144 raft_consensus.cc:504] T 00000000000000000000000000000000 P 351c72276fbd4fb692f8d3da58b72666 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "351c72276fbd4fb692f8d3da58b72666" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 40873 } }
I20260619 14:31:56.407639 2144 leader_election.cc:302] T 00000000000000000000000000000000 P 351c72276fbd4fb692f8d3da58b72666 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 351c72276fbd4fb692f8d3da58b72666; no voters:
I20260619 14:31:56.407995 2144 leader_election.cc:288] T 00000000000000000000000000000000 P 351c72276fbd4fb692f8d3da58b72666 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:31:56.408108 2149 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 351c72276fbd4fb692f8d3da58b72666 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:31:56.408445 2149 raft_consensus.cc:686] T 00000000000000000000000000000000 P 351c72276fbd4fb692f8d3da58b72666 [term 1 LEADER]: Becoming Leader. State: Replica: 351c72276fbd4fb692f8d3da58b72666, State: Running, Role: LEADER
I20260619 14:31:56.408833 2149 consensus_queue.cc:237] T 00000000000000000000000000000000 P 351c72276fbd4fb692f8d3da58b72666 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "351c72276fbd4fb692f8d3da58b72666" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 40873 } }
I20260619 14:31:56.409523 2144 sys_catalog.cc:564] T 00000000000000000000000000000000 P 351c72276fbd4fb692f8d3da58b72666 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:31:56.410825 2151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 351c72276fbd4fb692f8d3da58b72666 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 351c72276fbd4fb692f8d3da58b72666. Latest consensus state: current_term: 1 leader_uuid: "351c72276fbd4fb692f8d3da58b72666" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "351c72276fbd4fb692f8d3da58b72666" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 40873 } } }
I20260619 14:31:56.410825 2150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 351c72276fbd4fb692f8d3da58b72666 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "351c72276fbd4fb692f8d3da58b72666" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "351c72276fbd4fb692f8d3da58b72666" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 40873 } } }
I20260619 14:31:56.410995 2151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 351c72276fbd4fb692f8d3da58b72666 [sys.catalog]: This master's current role is: LEADER
I20260619 14:31:56.410995 2150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 351c72276fbd4fb692f8d3da58b72666 [sys.catalog]: This master's current role is: LEADER
I20260619 14:31:56.411384 2158 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Jun 19 14:31:56 dist-test-slave-70rm krb5kdc[2063](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:31:56.414937 2158 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:31:56.420182 2146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:31:56.393082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55156 (local address 127.2.2.62:40873)
0619 14:31:56.393573 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0619 14:31:56.393592 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 14:31:56.394810 (+ 1218us) server_negotiation.cc:407] Connection header received
0619 14:31:56.399306 (+ 4496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:31:56.399328 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:31:56.399770 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:31:56.400157 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:31:56.404118 (+ 3961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:31:56.405309 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:31:56.407227 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:31:56.407899 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:31:56.408113 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:31:56.413519 (+ 5406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:31:56.413527 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:31:56.413543 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:31:56.413580 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:31:56.415868 (+ 2288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:31:56.416421 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:31:56.416428 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:31:56.416434 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:31:56.416586 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:31:56.416912 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:31:56.416916 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:31:56.416917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:31:56.417490 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:31:56.417696 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0619 14:31:56.418445 (+ 749us) server_negotiation.cc:299] Negotiation successful
0619 14:31:56.418758 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":149,"threads_started":1}
I20260619 14:31:56.420683 2158 catalog_manager.cc:1347] Generated new cluster ID: 4824dadbd7554940a515a7804b7031f4
I20260619 14:31:56.420753 2158 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:31:56.438798 2158 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:31:56.439752 2158 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:31:56.451328 2158 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 351c72276fbd4fb692f8d3da58b72666: Generated new TSK 0
I20260619 14:31:56.452445 2158 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.2.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab.
Entry for principal kudu/127.2.2.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab.
I20260619 14:31:56.489852 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.1:0
--local_ip_for_outbound_sockets=127.2.2.1
--webserver_interface=127.2.2.1
--webserver_port=0
--tserver_master_addrs=127.2.2.62:40873
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab
--principal=kudu/127.2.2.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:45663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:31:56.615327 2170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:31:56.615648 2170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:31:56.615724 2170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:31:56.619976 2170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:31:56.620056 2170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:31:56.620121 2170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.1
I20260619 14:31:56.625484 2170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:45663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab
--principal=kudu/127.2.2.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.2.1
--webserver_port=0
--tserver_master_addrs=127.2.2.62:40873
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.1
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:31:56.626698 2170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:31:56.627784 2170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:31:56.637455 2178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:31:56.637466 2176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:31:56.637466 2175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:31:56.637765 2170 server_base.cc:1034] running on GCE node
I20260619 14:31:56.638330 2170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:31:56.638993 2170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:31:56.640182 2170 hybrid_clock.cc:648] HybridClock initialized: now 1781879516640157 us; error 41 us; skew 500 ppm
Jun 19 14:31:56 dist-test-slave-70rm krb5kdc[2063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879516, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:31:56.643121 2170 init.cc:377] Logged in from keytab as kudu/127.2.2.1@KRBTEST.COM (short username kudu)
I20260619 14:31:56.644029 2170 webserver.cc:460] Webserver started at http://127.2.2.1:38813/ using document root <none> and password file <none>
I20260619 14:31:56.644609 2170 fs_manager.cc:359] Metadata directory not provided
I20260619 14:31:56.644696 2170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:31:56.644922 2170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:31:56.647109 2170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/ts-0/data/instance:
uuid: "ca09b2644c8e48698748e2cf175cffda"
format_stamp: "Formatted at 2026-06-19 14:31:56 on dist-test-slave-70rm"
server_key: "5da3037685bd8c69d5dc6e5b9f85af14"
server_key_iv: "d858b8c808d9a0c3c317098f22757a7e"
server_key_version: "encryptionkey@0"
I20260619 14:31:56.647688 2170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/ts-0/wal/instance:
uuid: "ca09b2644c8e48698748e2cf175cffda"
format_stamp: "Formatted at 2026-06-19 14:31:56 on dist-test-slave-70rm"
server_key: "5da3037685bd8c69d5dc6e5b9f85af14"
server_key_iv: "d858b8c808d9a0c3c317098f22757a7e"
server_key_version: "encryptionkey@0"
I20260619 14:31:56.651531 2170 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:31:56.654104 2185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:31:56.655493 2170 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:31:56.655615 2170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/ts-0/wal
uuid: "ca09b2644c8e48698748e2cf175cffda"
format_stamp: "Formatted at 2026-06-19 14:31:56 on dist-test-slave-70rm"
server_key: "5da3037685bd8c69d5dc6e5b9f85af14"
server_key_iv: "d858b8c808d9a0c3c317098f22757a7e"
server_key_version: "encryptionkey@0"
I20260619 14:31:56.655740 2170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:31:56.665038 2170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:31:56.668228 2170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:31:56.668439 2170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:31:56.669031 2170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:31:56.669860 2170 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:31:56.669937 2170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:31:56.670007 2170 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:31:56.670053 2170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:31:56.679227 2170 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.1:33589
I20260619 14:31:56.679263 2298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.1:33589 every 8 connection(s)
I20260619 14:31:56.680199 2170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
I20260619 14:31:56.687074 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 2170
I20260619 14:31:56.687193 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/ts-0/wal/instance
I20260619 14:31:56.687444 2056 external_mini_cluster.cc:1428] Setting key 7789295caf97a643fff64471b5af853e
Jun 19 14:31:56 dist-test-slave-70rm krb5kdc[2063](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879516, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.1@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:31:56.696272 2146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:31:56.682275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:60723 (local address 127.2.2.62:40873)
0619 14:31:56.682413 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 14:31:56.682418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:31:56.683397 (+ 979us) server_negotiation.cc:407] Connection header received
0619 14:31:56.684887 (+ 1490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:31:56.684891 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:31:56.684967 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:31:56.685080 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:31:56.686535 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:31:56.687134 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:31:56.688648 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:31:56.688985 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:31:56.689129 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:31:56.692012 (+ 2883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:31:56.692015 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:31:56.692017 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:31:56.692042 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:31:56.693466 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:31:56.694265 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:31:56.694269 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:31:56.694270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:31:56.694439 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:31:56.694948 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:31:56.694950 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:31:56.694952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:31:56.695156 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:31:56.695241 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 14:31:56.695980 (+ 739us) server_negotiation.cc:299] Negotiation successful
0619 14:31:56.696120 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 14:31:56.697475 2301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:31:56.682674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:40873 (local address 127.2.2.1:60723)
0619 14:31:56.683210 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0619 14:31:56.683260 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 14:31:56.684628 (+ 1368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:31:56.685293 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:31:56.685300 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:31:56.685726 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:31:56.686345 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:31:56.686368 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:31:56.687339 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:31:56.687342 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:31:56.688494 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:31:56.688506 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:31:56.689113 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:31:56.689176 (+ 63us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:31:56.689363 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:31:56.690048 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:31:56.690066 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:31:56.691820 (+ 1754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:31:56.693595 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:31:56.693605 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:31:56.693628 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:31:56.694068 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:31:56.694572 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:31:56.694578 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:31:56.694583 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:31:56.694804 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:31:56.695267 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:31:56.695275 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:31:56.695655 (+ 380us) client_negotiation.cc:769] Sending connection context
0619 14:31:56.695885 (+ 230us) client_negotiation.cc:240] Negotiation successful
0619 14:31:56.696289 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":120,"threads_started":1}
WARNING: no policy specified for kudu/127.2.2.2@KRBTEST.COM; defaulting to no policy
I20260619 14:31:56.698881 2299 heartbeater.cc:347] Connected to a master server at 127.2.2.62:40873
I20260619 14:31:56.699165 2299 heartbeater.cc:464] Registering TS with master...
I20260619 14:31:56.699693 2299 heartbeater.cc:511] Master 127.2.2.62:40873 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.2@KRBTEST.COM" created.
I20260619 14:31:56.701484 2108 ts_manager.cc:194] Registered new tserver with Master: ca09b2644c8e48698748e2cf175cffda (127.2.2.1:33589)
I20260619 14:31:56.703199 2108 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.2.1@KRBTEST.COM'} at 127.2.2.1:60723
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab.
Entry for principal kudu/127.2.2.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab.
I20260619 14:31:56.717438 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.2:0
--local_ip_for_outbound_sockets=127.2.2.2
--webserver_interface=127.2.2.2
--webserver_port=0
--tserver_master_addrs=127.2.2.62:40873
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab
--principal=kudu/127.2.2.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:45663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:31:56.830983 2304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:31:56.831300 2304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:31:56.831377 2304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:31:56.835662 2304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:31:56.835750 2304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:31:56.835840 2304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.2
I20260619 14:31:56.841241 2304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:45663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab
--principal=kudu/127.2.2.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.2.2
--webserver_port=0
--tserver_master_addrs=127.2.2.62:40873
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.2
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:31:56.842434 2304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:31:56.843461 2304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:31:56.853313 2310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:31:56.853328 2304 server_base.cc:1034] running on GCE node
W20260619 14:31:56.853546 2312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:31:56.853425 2309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:31:56.854069 2304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:31:56.854872 2304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:31:56.856083 2304 hybrid_clock.cc:648] HybridClock initialized: now 1781879516856053 us; error 40 us; skew 500 ppm
Jun 19 14:31:56 dist-test-slave-70rm krb5kdc[2063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879516, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:31:56.859133 2304 init.cc:377] Logged in from keytab as kudu/127.2.2.2@KRBTEST.COM (short username kudu)
I20260619 14:31:56.860140 2304 webserver.cc:460] Webserver started at http://127.2.2.2:32981/ using document root <none> and password file <none>
I20260619 14:31:56.860718 2304 fs_manager.cc:359] Metadata directory not provided
I20260619 14:31:56.860807 2304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:31:56.861042 2304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:31:56.863250 2304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/ts-1/data/instance:
uuid: "42c90d3f4d144322abc1d5ff1aecbcd9"
format_stamp: "Formatted at 2026-06-19 14:31:56 on dist-test-slave-70rm"
server_key: "03afb4e469dcb31cb56183bd6a1938a9"
server_key_iv: "b6868c12beb854781e67b9f43663f16a"
server_key_version: "encryptionkey@0"
I20260619 14:31:56.863826 2304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/ts-1/wal/instance:
uuid: "42c90d3f4d144322abc1d5ff1aecbcd9"
format_stamp: "Formatted at 2026-06-19 14:31:56 on dist-test-slave-70rm"
server_key: "03afb4e469dcb31cb56183bd6a1938a9"
server_key_iv: "b6868c12beb854781e67b9f43663f16a"
server_key_version: "encryptionkey@0"
I20260619 14:31:56.867610 2304 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:31:56.870072 2319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:31:56.871345 2304 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:31:56.871476 2304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/ts-1/wal
uuid: "42c90d3f4d144322abc1d5ff1aecbcd9"
format_stamp: "Formatted at 2026-06-19 14:31:56 on dist-test-slave-70rm"
server_key: "03afb4e469dcb31cb56183bd6a1938a9"
server_key_iv: "b6868c12beb854781e67b9f43663f16a"
server_key_version: "encryptionkey@0"
I20260619 14:31:56.871589 2304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:31:56.883154 2304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:31:56.886530 2304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:31:56.886819 2304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:31:56.887876 2304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:31:56.888958 2304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:31:56.889021 2304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:31:56.889092 2304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:31:56.889134 2304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:31:56.899302 2304 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.2:40949
I20260619 14:31:56.899348 2432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.2:40949 every 8 connection(s)
I20260619 14:31:56.900278 2304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
I20260619 14:31:56.903972 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 2304
I20260619 14:31:56.904119 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/ts-1/wal/instance
I20260619 14:31:56.904409 2056 external_mini_cluster.cc:1428] Setting key 29859ece43f699369f4ba99740331283
Jun 19 14:31:56 dist-test-slave-70rm krb5kdc[2063](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879516, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.2@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.2.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.3@KRBTEST.COM" created.
I20260619 14:31:56.916070 2146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:31:56.902210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:57615 (local address 127.2.2.62:40873)
0619 14:31:56.902352 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 14:31:56.902355 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:31:56.903235 (+ 880us) server_negotiation.cc:407] Connection header received
0619 14:31:56.905052 (+ 1817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:31:56.905055 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:31:56.905101 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:31:56.905199 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:31:56.906592 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:31:56.907130 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:31:56.908310 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:31:56.908535 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:31:56.908646 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:31:56.911858 (+ 3212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:31:56.911862 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:31:56.911870 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:31:56.911898 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:31:56.913594 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:31:56.914247 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:31:56.914251 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:31:56.914253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:31:56.914343 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:31:56.914789 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:31:56.914792 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:31:56.914794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:31:56.915005 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:31:56.915114 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 14:31:56.915743 (+ 629us) server_negotiation.cc:299] Negotiation successful
0619 14:31:56.915878 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 14:31:56.917205 2435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:31:56.902604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:40873 (local address 127.2.2.2:57615)
0619 14:31:56.903077 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0619 14:31:56.903120 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 14:31:56.904788 (+ 1668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:31:56.905380 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:31:56.905387 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:31:56.905869 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:31:56.906435 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:31:56.906449 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:31:56.907255 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:31:56.907264 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:31:56.908166 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:31:56.908176 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:31:56.908631 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:31:56.908693 (+ 62us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:31:56.908871 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:31:56.909590 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:31:56.909616 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:31:56.911656 (+ 2040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:31:56.913747 (+ 2091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:31:56.913754 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:31:56.913771 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:31:56.914092 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:31:56.914457 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:31:56.914461 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:31:56.914474 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:31:56.914684 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:31:56.915125 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:31:56.915134 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:31:56.915471 (+ 337us) client_negotiation.cc:769] Sending connection context
0619 14:31:56.915683 (+ 212us) client_negotiation.cc:240] Negotiation successful
0619 14:31:56.916052 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":119,"threads_started":1}
I20260619 14:31:56.918588 2433 heartbeater.cc:347] Connected to a master server at 127.2.2.62:40873
I20260619 14:31:56.918895 2433 heartbeater.cc:464] Registering TS with master...
I20260619 14:31:56.919514 2433 heartbeater.cc:511] Master 127.2.2.62:40873 requested a full tablet report, sending...
I20260619 14:31:56.920768 2108 ts_manager.cc:194] Registered new tserver with Master: 42c90d3f4d144322abc1d5ff1aecbcd9 (127.2.2.2:40949)
I20260619 14:31:56.921769 2108 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.2.2@KRBTEST.COM'} at 127.2.2.2:57615
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab.
Entry for principal kudu/127.2.2.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab.
I20260619 14:31:56.932891 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.3:0
--local_ip_for_outbound_sockets=127.2.2.3
--webserver_interface=127.2.2.3
--webserver_port=0
--tserver_master_addrs=127.2.2.62:40873
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab
--principal=kudu/127.2.2.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:45663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:31:57.045760 2438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:31:57.046036 2438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:31:57.046082 2438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:31:57.050573 2438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:31:57.050639 2438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:31:57.050714 2438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.3
I20260619 14:31:57.056185 2438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:45663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab
--principal=kudu/127.2.2.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.2.3
--webserver_port=0
--tserver_master_addrs=127.2.2.62:40873
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.3
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:31:57.057207 2438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:31:57.058224 2438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:31:57.067680 2443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:31:57.067718 2438 server_base.cc:1034] running on GCE node
W20260619 14:31:57.067682 2444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:31:57.067680 2446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:31:57.068456 2438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:31:57.069054 2438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:31:57.070214 2438 hybrid_clock.cc:648] HybridClock initialized: now 1781879517070197 us; error 26 us; skew 500 ppm
Jun 19 14:31:57 dist-test-slave-70rm krb5kdc[2063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879517, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:31:57.073460 2438 init.cc:377] Logged in from keytab as kudu/127.2.2.3@KRBTEST.COM (short username kudu)
I20260619 14:31:57.074424 2438 webserver.cc:460] Webserver started at http://127.2.2.3:33397/ using document root <none> and password file <none>
I20260619 14:31:57.074991 2438 fs_manager.cc:359] Metadata directory not provided
I20260619 14:31:57.075055 2438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:31:57.075222 2438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:31:57.077365 2438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/ts-2/data/instance:
uuid: "5e874068994b48adb12329b5abdcb1b8"
format_stamp: "Formatted at 2026-06-19 14:31:57 on dist-test-slave-70rm"
server_key: "7f7bb06b9dd3420049bcbf8e2f2f8779"
server_key_iv: "c32910be711e34b4dc9aa8dd39b984dd"
server_key_version: "encryptionkey@0"
I20260619 14:31:57.077913 2438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/ts-2/wal/instance:
uuid: "5e874068994b48adb12329b5abdcb1b8"
format_stamp: "Formatted at 2026-06-19 14:31:57 on dist-test-slave-70rm"
server_key: "7f7bb06b9dd3420049bcbf8e2f2f8779"
server_key_iv: "c32910be711e34b4dc9aa8dd39b984dd"
server_key_version: "encryptionkey@0"
I20260619 14:31:57.081847 2438 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 14:31:57.084546 2453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:31:57.085937 2438 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:31:57.086068 2438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/ts-2/wal
uuid: "5e874068994b48adb12329b5abdcb1b8"
format_stamp: "Formatted at 2026-06-19 14:31:57 on dist-test-slave-70rm"
server_key: "7f7bb06b9dd3420049bcbf8e2f2f8779"
server_key_iv: "c32910be711e34b4dc9aa8dd39b984dd"
server_key_version: "encryptionkey@0"
I20260619 14:31:57.086256 2438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:31:57.098860 2438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:31:57.102573 2438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:31:57.102742 2438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:31:57.103318 2438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:31:57.104174 2438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:31:57.104224 2438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:31:57.104269 2438 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:31:57.104286 2438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:31:57.115470 2438 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.3:45369
I20260619 14:31:57.115497 2566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.3:45369 every 8 connection(s)
I20260619 14:31:57.116477 2438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
I20260619 14:31:57.119278 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 2438
I20260619 14:31:57.119410 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879514993456-2056-0/minicluster-data/ts-2/wal/instance
I20260619 14:31:57.119701 2056 external_mini_cluster.cc:1428] Setting key 55519a41b7f9682a639695a40505ad53
Jun 19 14:31:57 dist-test-slave-70rm krb5kdc[2063](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879517, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.3@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:31:57.131232 2146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:31:57.118483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.3:46285 (local address 127.2.2.62:40873)
0619 14:31:57.118637 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 14:31:57.118642 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:31:57.119731 (+ 1089us) server_negotiation.cc:407] Connection header received
0619 14:31:57.121499 (+ 1768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:31:57.121503 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:31:57.121554 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:31:57.121635 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:31:57.123030 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:31:57.123547 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:31:57.124772 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:31:57.124980 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:31:57.125059 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:31:57.127768 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:31:57.127772 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:31:57.127774 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:31:57.127801 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:31:57.129195 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:31:57.129747 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:31:57.129750 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:31:57.129751 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:31:57.129823 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:31:57.130229 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:31:57.130232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:31:57.130233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:31:57.130403 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:31:57.130480 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 14:31:57.130991 (+ 511us) server_negotiation.cc:299] Negotiation successful
0619 14:31:57.131109 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 14:31:57.132134 2569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:31:57.118996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:40873 (local address 127.2.2.3:46285)
0619 14:31:57.119529 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0619 14:31:57.119602 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0619 14:31:57.121253 (+ 1651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:31:57.121794 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:31:57.121802 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:31:57.122353 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:31:57.122890 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:31:57.122905 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:31:57.123655 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:31:57.123658 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:31:57.124660 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:31:57.124669 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:31:57.125065 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:31:57.125114 (+ 49us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:31:57.125269 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:31:57.125964 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:31:57.125990 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:31:57.127599 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:31:57.129296 (+ 1697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:31:57.129301 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:31:57.129315 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:31:57.129662 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:31:57.129898 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:31:57.129900 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:31:57.129901 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:31:57.130073 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:31:57.130493 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:31:57.130499 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:31:57.130763 (+ 264us) client_negotiation.cc:769] Sending connection context
0619 14:31:57.130929 (+ 166us) client_negotiation.cc:240] Negotiation successful
0619 14:31:57.131232 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":146,"threads_started":1}
I20260619 14:31:57.133363 2567 heartbeater.cc:347] Connected to a master server at 127.2.2.62:40873
I20260619 14:31:57.133622 2567 heartbeater.cc:464] Registering TS with master...
I20260619 14:31:57.134322 2567 heartbeater.cc:511] Master 127.2.2.62:40873 requested a full tablet report, sending...
I20260619 14:31:57.135447 2108 ts_manager.cc:194] Registered new tserver with Master: 5e874068994b48adb12329b5abdcb1b8 (127.2.2.3:45369)
I20260619 14:31:57.136051 2108 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.2.3@KRBTEST.COM'} at 127.2.2.3:46285
I20260619 14:31:57.146047 2056 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 14:31:57 dist-test-slave-70rm krb5kdc[2063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879517, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 14:31:57 dist-test-slave-70rm krb5kdc[2063](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879517, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.2.1@KRBTEST.COM
I20260619 14:31:57.172369 2574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:31:57.162907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60532 (local address 127.2.2.1:33589)
0619 14:31:57.163243 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0619 14:31:57.163258 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:31:57.163285 (+ 27us) server_negotiation.cc:407] Connection header received
0619 14:31:57.163414 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:31:57.163421 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:31:57.163632 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:31:57.163789 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:31:57.164469 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:31:57.165244 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:31:57.166114 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:31:57.166539 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:31:57.166662 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:31:57.169160 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:31:57.169170 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:31:57.169183 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:31:57.169211 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:31:57.170858 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:31:57.171251 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:31:57.171256 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:31:57.171257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:31:57.171337 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:31:57.171548 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:31:57.171550 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:31:57.171551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:31:57.171793 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:31:57.171966 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 14:31:57.172090 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 14:31:57.172214 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":113,"threads_started":1}
W20260619 14:31:57.173365 2213 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60532
Jun 19 14:31:57 dist-test-slave-70rm krb5kdc[2063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 14:31:57 dist-test-slave-70rm krb5kdc[2063](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.2.1@KRBTEST.COM
I20260619 14:31:57.199818 2574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:31:57.191522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60534 (local address 127.2.2.1:33589)
0619 14:31:57.191684 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 14:31:57.191688 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:31:57.191784 (+ 96us) server_negotiation.cc:407] Connection header received
0619 14:31:57.191919 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:31:57.191922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:31:57.191962 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:31:57.192035 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:31:57.192754 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:31:57.193271 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:31:57.194095 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:31:57.194347 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:31:57.194425 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:31:57.196817 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:31:57.196820 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:31:57.196822 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:31:57.196845 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:31:57.198205 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:31:57.198604 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:31:57.198606 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:31:57.198608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:31:57.198688 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:31:57.198993 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:31:57.198995 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:31:57.198996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:31:57.199172 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:31:57.199272 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 14:31:57.199516 (+ 244us) server_negotiation.cc:299] Negotiation successful
0619 14:31:57.199623 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 14:31:57.200940 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 2170
I20260619 14:31:57.206670 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 2304
I20260619 14:31:57.213788 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 2438
I20260619 14:31:57.219329 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 2076
2026-06-19T14:31:57Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2220 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:31:57 dist-test-slave-70rm krb5kdc[2581](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:31:57 dist-test-slave-70rm krb5kdc[2581](info): set up 2 sockets
Jun 19 14:31:57 dist-test-slave-70rm krb5kdc[2581](info): commencing operation
krb5kdc: starting...
W20260619 14:31:58.265780 2056 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Jun 19 14:31:58 dist-test-slave-70rm krb5kdc[2581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:31:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:31:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.2.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab.
Entry for principal kudu/127.2.2.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab.
I20260619 14:31:58.388895 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:46197
--webserver_interface=127.2.2.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:44135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.2.62:46197
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:31:58.498240 2594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:31:58.498641 2594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:31:58.498687 2594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:31:58.502699 2594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:31:58.502753 2594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:31:58.502774 2594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:31:58.502790 2594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:31:58.502805 2594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:31:58.508090 2594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:44135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.2.62:46197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:46197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.2.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:31:58.509097 2594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:31:58.510057 2594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:31:58.518230 2594 server_base.cc:1034] running on GCE node
W20260619 14:31:58.518181 2599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:31:58.518210 2600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:31:58.518174 2602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:31:58.518973 2594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:31:58.520036 2594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:31:58.521209 2594 hybrid_clock.cc:648] HybridClock initialized: now 1781879518521198 us; error 29 us; skew 500 ppm
Jun 19 14:31:58 dist-test-slave-70rm krb5kdc[2581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879518, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:31:58.523850 2594 init.cc:377] Logged in from keytab as kudu/127.2.2.62@KRBTEST.COM (short username kudu)
I20260619 14:31:58.524749 2594 webserver.cc:460] Webserver started at http://127.2.2.62:37349/ using document root <none> and password file <none>
I20260619 14:31:58.525300 2594 fs_manager.cc:359] Metadata directory not provided
I20260619 14:31:58.525383 2594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:31:58.525607 2594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:31:58.527695 2594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/master-0/data/instance:
uuid: "2ddda61d347445a1ae13e05c590584d7"
format_stamp: "Formatted at 2026-06-19 14:31:58 on dist-test-slave-70rm"
server_key: "50c97a19e2761328df5fc5823acf680a"
server_key_iv: "3d9e00c5d8ef3ba9deabf09c8f5e5c7e"
server_key_version: "encryptionkey@0"
I20260619 14:31:58.528266 2594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/master-0/wal/instance:
uuid: "2ddda61d347445a1ae13e05c590584d7"
format_stamp: "Formatted at 2026-06-19 14:31:58 on dist-test-slave-70rm"
server_key: "50c97a19e2761328df5fc5823acf680a"
server_key_iv: "3d9e00c5d8ef3ba9deabf09c8f5e5c7e"
server_key_version: "encryptionkey@0"
I20260619 14:31:58.532066 2594 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260619 14:31:58.534475 2609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:31:58.535713 2594 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 14:31:58.535853 2594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/master-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/master-0/wal
uuid: "2ddda61d347445a1ae13e05c590584d7"
format_stamp: "Formatted at 2026-06-19 14:31:58 on dist-test-slave-70rm"
server_key: "50c97a19e2761328df5fc5823acf680a"
server_key_iv: "3d9e00c5d8ef3ba9deabf09c8f5e5c7e"
server_key_version: "encryptionkey@0"
I20260619 14:31:58.535972 2594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:31:58.558923 2594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:31:58.562096 2594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:31:58.562328 2594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:31:58.569453 2594 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.62:46197
I20260619 14:31:58.569500 2661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.62:46197 every 8 connection(s)
I20260619 14:31:58.570472 2594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
I20260619 14:31:58.573235 2662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:31:58.575040 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 2594
I20260619 14:31:58.575152 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/master-0/wal/instance
I20260619 14:31:58.575415 2056 external_mini_cluster.cc:1428] Setting key 7ae35033c85c3902f575efa810e54220
I20260619 14:31:58.579792 2662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ddda61d347445a1ae13e05c590584d7: Bootstrap starting.
I20260619 14:31:58.582449 2662 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ddda61d347445a1ae13e05c590584d7: Neither blocks nor log segments found. Creating new log.
I20260619 14:31:58.583345 2662 log.cc:824] T 00000000000000000000000000000000 P 2ddda61d347445a1ae13e05c590584d7: Log is configured to *not* fsync() on all Append() calls
Jun 19 14:31:58 dist-test-slave-70rm krb5kdc[2581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:31:58.585572 2662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ddda61d347445a1ae13e05c590584d7: No bootstrap required, opened a new log
I20260619 14:31:58.589071 2662 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ddda61d347445a1ae13e05c590584d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ddda61d347445a1ae13e05c590584d7" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 46197 } }
I20260619 14:31:58.589311 2662 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ddda61d347445a1ae13e05c590584d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:31:58.589401 2662 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ddda61d347445a1ae13e05c590584d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ddda61d347445a1ae13e05c590584d7, State: Initialized, Role: FOLLOWER
I20260619 14:31:58.589794 2665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:31:58.576756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41554 (local address 127.2.2.62:46197)
0619 14:31:58.577155 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0619 14:31:58.577171 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 14:31:58.577204 (+ 33us) server_negotiation.cc:407] Connection header received
0619 14:31:58.577812 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:31:58.577829 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:31:58.578229 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:31:58.578513 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:31:58.579389 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:31:58.580584 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:31:58.581746 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:31:58.582287 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:31:58.582405 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:31:58.585111 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:31:58.585117 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:31:58.585134 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:31:58.585164 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:31:58.586915 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:31:58.587384 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:31:58.587389 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:31:58.587393 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:31:58.587500 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:31:58.587777 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:31:58.587780 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:31:58.587782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:31:58.588162 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:31:58.588369 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0619 14:31:58.588670 (+ 301us) server_negotiation.cc:299] Negotiation successful
0619 14:31:58.588937 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":115,"threads_started":1}
I20260619 14:31:58.589862 2662 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ddda61d347445a1ae13e05c590584d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ddda61d347445a1ae13e05c590584d7" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 46197 } }
I20260619 14:31:58.590047 2662 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ddda61d347445a1ae13e05c590584d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:31:58.590144 2662 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ddda61d347445a1ae13e05c590584d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:31:58.590246 2662 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ddda61d347445a1ae13e05c590584d7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:31:58.591171 2662 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ddda61d347445a1ae13e05c590584d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ddda61d347445a1ae13e05c590584d7" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 46197 } }
I20260619 14:31:58.591514 2662 leader_election.cc:302] T 00000000000000000000000000000000 P 2ddda61d347445a1ae13e05c590584d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ddda61d347445a1ae13e05c590584d7; no voters:
I20260619 14:31:58.591854 2662 leader_election.cc:288] T 00000000000000000000000000000000 P 2ddda61d347445a1ae13e05c590584d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:31:58.591907 2667 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ddda61d347445a1ae13e05c590584d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:31:58.592202 2667 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ddda61d347445a1ae13e05c590584d7 [term 1 LEADER]: Becoming Leader. State: Replica: 2ddda61d347445a1ae13e05c590584d7, State: Running, Role: LEADER
I20260619 14:31:58.592494 2667 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ddda61d347445a1ae13e05c590584d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ddda61d347445a1ae13e05c590584d7" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 46197 } }
I20260619 14:31:58.593214 2662 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ddda61d347445a1ae13e05c590584d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:31:58.593638 2667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ddda61d347445a1ae13e05c590584d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ddda61d347445a1ae13e05c590584d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ddda61d347445a1ae13e05c590584d7" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 46197 } } }
I20260619 14:31:58.593755 2667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ddda61d347445a1ae13e05c590584d7 [sys.catalog]: This master's current role is: LEADER
I20260619 14:31:58.594046 2668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ddda61d347445a1ae13e05c590584d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ddda61d347445a1ae13e05c590584d7. Latest consensus state: current_term: 1 leader_uuid: "2ddda61d347445a1ae13e05c590584d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ddda61d347445a1ae13e05c590584d7" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 46197 } } }
I20260619 14:31:58.594197 2668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ddda61d347445a1ae13e05c590584d7 [sys.catalog]: This master's current role is: LEADER
W20260619 14:31:58.597805 2682 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2ddda61d347445a1ae13e05c590584d7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:31:58.597904 2682 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:31:58.597985 2671 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:31:58.598865 2671 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:31:58.604480 2671 catalog_manager.cc:1347] Generated new cluster ID: cd467b3c2b46461386ec901474616c76
I20260619 14:31:58.604557 2671 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:31:58.618562 2671 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:31:58.619911 2671 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:31:58.627782 2671 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ddda61d347445a1ae13e05c590584d7: Generated new TSK 0
I20260619 14:31:58.628607 2671 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.2.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab.
Entry for principal kudu/127.2.2.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab.
I20260619 14:31:58.657100 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.1:0
--local_ip_for_outbound_sockets=127.2.2.1
--webserver_interface=127.2.2.1
--webserver_port=0
--tserver_master_addrs=127.2.2.62:46197
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab
--principal=kudu/127.2.2.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:44135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:31:58.770481 2688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:31:58.770768 2688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:31:58.770814 2688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:31:58.774967 2688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:31:58.775023 2688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:31:58.775120 2688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.1
I20260619 14:31:58.780416 2688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:44135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab
--principal=kudu/127.2.2.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.2.1
--webserver_port=0
--tserver_master_addrs=127.2.2.62:46197
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.1
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:31:58.781520 2688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:31:58.782655 2688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:31:58.792438 2694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:31:58.792379 2696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:31:58.792392 2693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:31:58.792845 2688 server_base.cc:1034] running on GCE node
I20260619 14:31:58.793310 2688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:31:58.794015 2688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:31:58.795210 2688 hybrid_clock.cc:648] HybridClock initialized: now 1781879518795169 us; error 62 us; skew 500 ppm
Jun 19 14:31:58 dist-test-slave-70rm krb5kdc[2581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879518, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:31:58.798182 2688 init.cc:377] Logged in from keytab as kudu/127.2.2.1@KRBTEST.COM (short username kudu)
I20260619 14:31:58.799203 2688 webserver.cc:460] Webserver started at http://127.2.2.1:40653/ using document root <none> and password file <none>
I20260619 14:31:58.799741 2688 fs_manager.cc:359] Metadata directory not provided
I20260619 14:31:58.799798 2688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:31:58.799960 2688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:31:58.801950 2688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/ts-0/data/instance:
uuid: "a80837283a8645d49598f3fe3dbdb2b6"
format_stamp: "Formatted at 2026-06-19 14:31:58 on dist-test-slave-70rm"
server_key: "04ee0962426121b4c6d2da908d77217d"
server_key_iv: "624e87826ab7493f76d885515aeafc84"
server_key_version: "encryptionkey@0"
I20260619 14:31:58.802552 2688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/ts-0/wal/instance:
uuid: "a80837283a8645d49598f3fe3dbdb2b6"
format_stamp: "Formatted at 2026-06-19 14:31:58 on dist-test-slave-70rm"
server_key: "04ee0962426121b4c6d2da908d77217d"
server_key_iv: "624e87826ab7493f76d885515aeafc84"
server_key_version: "encryptionkey@0"
I20260619 14:31:58.806433 2688 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 14:31:58.808825 2703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:31:58.810029 2688 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:31:58.810209 2688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/ts-0/wal
uuid: "a80837283a8645d49598f3fe3dbdb2b6"
format_stamp: "Formatted at 2026-06-19 14:31:58 on dist-test-slave-70rm"
server_key: "04ee0962426121b4c6d2da908d77217d"
server_key_iv: "624e87826ab7493f76d885515aeafc84"
server_key_version: "encryptionkey@0"
I20260619 14:31:58.810305 2688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:31:58.820940 2688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:31:58.823941 2688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:31:58.824165 2688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:31:58.824734 2688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:31:58.825536 2688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:31:58.825603 2688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:31:58.825670 2688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:31:58.825714 2688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:31:58.834741 2688 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.1:41169
I20260619 14:31:58.834786 2816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.1:41169 every 8 connection(s)
I20260619 14:31:58.835696 2688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
I20260619 14:31:58.843415 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 2688
I20260619 14:31:58.843506 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/ts-0/wal/instance
I20260619 14:31:58.843787 2056 external_mini_cluster.cc:1428] Setting key 2ec42348684b0b9eecf8f0baa75d0b57
Jun 19 14:31:58 dist-test-slave-70rm krb5kdc[2581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879518, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.1@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:31:58.850467 2665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:31:58.837516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:47211 (local address 127.2.2.62:46197)
0619 14:31:58.837651 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 14:31:58.837655 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:31:58.838541 (+ 886us) server_negotiation.cc:407] Connection header received
0619 14:31:58.840053 (+ 1512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:31:58.840056 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:31:58.840105 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:31:58.840195 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:31:58.841433 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:31:58.841983 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:31:58.843220 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:31:58.843447 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:31:58.843585 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:31:58.846439 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:31:58.846442 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:31:58.846444 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:31:58.846467 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:31:58.848301 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:31:58.848851 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:31:58.848854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:31:58.848856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:31:58.848949 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:31:58.849293 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:31:58.849296 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:31:58.849298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:31:58.849499 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:31:58.849621 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 14:31:58.850150 (+ 529us) server_negotiation.cc:299] Negotiation successful
0619 14:31:58.850282 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 14:31:58.851398 2819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:31:58.837899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46197 (local address 127.2.2.1:47211)
0619 14:31:58.838375 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0619 14:31:58.838417 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 14:31:58.839856 (+ 1439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:31:58.840418 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:31:58.840425 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:31:58.840783 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:31:58.841278 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:31:58.841291 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:31:58.842137 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:31:58.842140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:31:58.843098 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:31:58.843107 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:31:58.843600 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:31:58.843661 (+ 61us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:31:58.843878 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:31:58.844500 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:31:58.844523 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:31:58.846271 (+ 1748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:31:58.848449 (+ 2178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:31:58.848455 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:31:58.848471 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:31:58.848742 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:31:58.849038 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:31:58.849042 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:31:58.849044 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:31:58.849195 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:31:58.849621 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:31:58.849629 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:31:58.849888 (+ 259us) client_negotiation.cc:769] Sending connection context
0619 14:31:58.850076 (+ 188us) client_negotiation.cc:240] Negotiation successful
0619 14:31:58.850448 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":170,"threads_started":1}
I20260619 14:31:58.852818 2817 heartbeater.cc:347] Connected to a master server at 127.2.2.62:46197
I20260619 14:31:58.853109 2817 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.2.2.2@KRBTEST.COM; defaulting to no policy
I20260619 14:31:58.853644 2817 heartbeater.cc:511] Master 127.2.2.62:46197 requested a full tablet report, sending...
I20260619 14:31:58.855264 2624 ts_manager.cc:194] Registered new tserver with Master: a80837283a8645d49598f3fe3dbdb2b6 (127.2.2.1:41169)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.2@KRBTEST.COM" created.
I20260619 14:31:58.856437 2624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.2.1@KRBTEST.COM'} at 127.2.2.1:47211
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab.
Entry for principal kudu/127.2.2.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab.
I20260619 14:31:58.871279 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.2:0
--local_ip_for_outbound_sockets=127.2.2.2
--webserver_interface=127.2.2.2
--webserver_port=0
--tserver_master_addrs=127.2.2.62:46197
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab
--principal=kudu/127.2.2.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:44135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:31:58.984436 2822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:31:58.984709 2822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:31:58.984753 2822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:31:58.989153 2822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:31:58.989218 2822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:31:58.989279 2822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.2
I20260619 14:31:58.994724 2822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:44135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab
--principal=kudu/127.2.2.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.2.2
--webserver_port=0
--tserver_master_addrs=127.2.2.62:46197
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.2
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:31:58.995802 2822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:31:58.996848 2822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:31:59.006279 2827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:31:59.006278 2830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:31:59.006278 2828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:31:59.006644 2822 server_base.cc:1034] running on GCE node
I20260619 14:31:59.007081 2822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:31:59.007737 2822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:31:59.008910 2822 hybrid_clock.cc:648] HybridClock initialized: now 1781879519008882 us; error 50 us; skew 500 ppm
Jun 19 14:31:59 dist-test-slave-70rm krb5kdc[2581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879519, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:31:59.011601 2822 init.cc:377] Logged in from keytab as kudu/127.2.2.2@KRBTEST.COM (short username kudu)
I20260619 14:31:59.012439 2822 webserver.cc:460] Webserver started at http://127.2.2.2:42065/ using document root <none> and password file <none>
I20260619 14:31:59.013033 2822 fs_manager.cc:359] Metadata directory not provided
I20260619 14:31:59.013119 2822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:31:59.013336 2822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:31:59.015475 2822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/ts-1/data/instance:
uuid: "0d5edcea117b436c9cc03ea3a0af7c09"
format_stamp: "Formatted at 2026-06-19 14:31:59 on dist-test-slave-70rm"
server_key: "513d886a8d3ac4ecbf8f4282ec839dc0"
server_key_iv: "c81e7f043387ea36ed3be493edf10510"
server_key_version: "encryptionkey@0"
I20260619 14:31:59.016037 2822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/ts-1/wal/instance:
uuid: "0d5edcea117b436c9cc03ea3a0af7c09"
format_stamp: "Formatted at 2026-06-19 14:31:59 on dist-test-slave-70rm"
server_key: "513d886a8d3ac4ecbf8f4282ec839dc0"
server_key_iv: "c81e7f043387ea36ed3be493edf10510"
server_key_version: "encryptionkey@0"
I20260619 14:31:59.019605 2822 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:31:59.021855 2837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:31:59.023051 2822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:31:59.023181 2822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/ts-1/wal
uuid: "0d5edcea117b436c9cc03ea3a0af7c09"
format_stamp: "Formatted at 2026-06-19 14:31:59 on dist-test-slave-70rm"
server_key: "513d886a8d3ac4ecbf8f4282ec839dc0"
server_key_iv: "c81e7f043387ea36ed3be493edf10510"
server_key_version: "encryptionkey@0"
I20260619 14:31:59.023284 2822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:31:59.033111 2822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:31:59.035910 2822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:31:59.036120 2822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:31:59.036680 2822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:31:59.037469 2822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:31:59.037544 2822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:31:59.037612 2822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:31:59.037649 2822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:31:59.046284 2822 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.2:36285
I20260619 14:31:59.046305 2950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.2:36285 every 8 connection(s)
I20260619 14:31:59.047147 2822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
I20260619 14:31:59.047456 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 2822
I20260619 14:31:59.047576 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/ts-1/wal/instance
I20260619 14:31:59.047834 2056 external_mini_cluster.cc:1428] Setting key 7b17a240a710eec695a568a8c6a9b7ea
WARNING: no policy specified for kudu/127.2.2.3@KRBTEST.COM; defaulting to no policy
Jun 19 14:31:59 dist-test-slave-70rm krb5kdc[2581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879519, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.2@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.3@KRBTEST.COM" created.
I20260619 14:31:59.060942 2665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:31:59.049011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:38947 (local address 127.2.2.62:46197)
0619 14:31:59.049149 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 14:31:59.049155 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:31:59.050044 (+ 889us) server_negotiation.cc:407] Connection header received
0619 14:31:59.051419 (+ 1375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:31:59.051424 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:31:59.051492 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:31:59.051600 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:31:59.052722 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:31:59.053238 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:31:59.054441 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:31:59.054657 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:31:59.054761 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:31:59.057172 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:31:59.057175 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:31:59.057176 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:31:59.057199 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:31:59.058681 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:31:59.059301 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:31:59.059305 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:31:59.059306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:31:59.059416 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:31:59.059793 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:31:59.059800 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:31:59.059805 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:31:59.060065 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:31:59.060174 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 14:31:59.060647 (+ 473us) server_negotiation.cc:299] Negotiation successful
0619 14:31:59.060780 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 14:31:59.061836 2953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:31:59.049439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46197 (local address 127.2.2.2:38947)
0619 14:31:59.049887 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0619 14:31:59.049937 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 14:31:59.051218 (+ 1281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:31:59.051749 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:31:59.051756 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:31:59.052094 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:31:59.052563 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:31:59.052582 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:31:59.053358 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:31:59.053362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:31:59.054324 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:31:59.054333 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:31:59.054757 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:31:59.054811 (+ 54us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:31:59.054976 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:31:59.055583 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:31:59.055600 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:31:59.056970 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:31:59.058797 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:31:59.058803 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:31:59.058820 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:31:59.059202 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:31:59.059544 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:31:59.059546 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:31:59.059548 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:31:59.059702 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:31:59.060173 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:31:59.060177 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:31:59.060410 (+ 233us) client_negotiation.cc:769] Sending connection context
0619 14:31:59.060613 (+ 203us) client_negotiation.cc:240] Negotiation successful
0619 14:31:59.060941 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":133,"threads_started":1}
I20260619 14:31:59.062978 2951 heartbeater.cc:347] Connected to a master server at 127.2.2.62:46197
I20260619 14:31:59.063232 2951 heartbeater.cc:464] Registering TS with master...
I20260619 14:31:59.063699 2951 heartbeater.cc:511] Master 127.2.2.62:46197 requested a full tablet report, sending...
I20260619 14:31:59.064667 2624 ts_manager.cc:194] Registered new tserver with Master: 0d5edcea117b436c9cc03ea3a0af7c09 (127.2.2.2:36285)
I20260619 14:31:59.065178 2624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.2.2@KRBTEST.COM'} at 127.2.2.2:38947
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab.
Entry for principal kudu/127.2.2.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab.
I20260619 14:31:59.074872 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.3:0
--local_ip_for_outbound_sockets=127.2.2.3
--webserver_interface=127.2.2.3
--webserver_port=0
--tserver_master_addrs=127.2.2.62:46197
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab
--principal=kudu/127.2.2.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:44135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:31:59.185142 2956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:31:59.185420 2956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:31:59.185465 2956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:31:59.189544 2956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:31:59.189641 2956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:31:59.189755 2956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.3
I20260619 14:31:59.195338 2956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:44135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab
--principal=kudu/127.2.2.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.2.3
--webserver_port=0
--tserver_master_addrs=127.2.2.62:46197
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.3
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:31:59.196353 2956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:31:59.197312 2956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:31:59.206673 2956 server_base.cc:1034] running on GCE node
W20260619 14:31:59.206622 2962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:31:59.206609 2964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:31:59.206621 2961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:31:59.207273 2956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:31:59.207944 2956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:31:59.209107 2956 hybrid_clock.cc:648] HybridClock initialized: now 1781879519209087 us; error 31 us; skew 500 ppm
Jun 19 14:31:59 dist-test-slave-70rm krb5kdc[2581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879519, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:31:59.212059 2956 init.cc:377] Logged in from keytab as kudu/127.2.2.3@KRBTEST.COM (short username kudu)
I20260619 14:31:59.212934 2956 webserver.cc:460] Webserver started at http://127.2.2.3:34825/ using document root <none> and password file <none>
I20260619 14:31:59.213490 2956 fs_manager.cc:359] Metadata directory not provided
I20260619 14:31:59.213603 2956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:31:59.213814 2956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:31:59.215893 2956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/ts-2/data/instance:
uuid: "80ec3f91128c49f09d7f0ef733624c65"
format_stamp: "Formatted at 2026-06-19 14:31:59 on dist-test-slave-70rm"
server_key: "cb236c4778bcc6adb2bb6301bf9f7fcf"
server_key_iv: "726579d83f1d671730ad0c323afd3651"
server_key_version: "encryptionkey@0"
I20260619 14:31:59.216461 2956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/ts-2/wal/instance:
uuid: "80ec3f91128c49f09d7f0ef733624c65"
format_stamp: "Formatted at 2026-06-19 14:31:59 on dist-test-slave-70rm"
server_key: "cb236c4778bcc6adb2bb6301bf9f7fcf"
server_key_iv: "726579d83f1d671730ad0c323afd3651"
server_key_version: "encryptionkey@0"
I20260619 14:31:59.220207 2956 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:31:59.222572 2971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:31:59.223814 2956 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:31:59.223984 2956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/ts-2/wal
uuid: "80ec3f91128c49f09d7f0ef733624c65"
format_stamp: "Formatted at 2026-06-19 14:31:59 on dist-test-slave-70rm"
server_key: "cb236c4778bcc6adb2bb6301bf9f7fcf"
server_key_iv: "726579d83f1d671730ad0c323afd3651"
server_key_version: "encryptionkey@0"
I20260619 14:31:59.224103 2956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:31:59.248685 2956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:31:59.251696 2956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:31:59.251894 2956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:31:59.252467 2956 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:31:59.253269 2956 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:31:59.253337 2956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:31:59.253404 2956 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:31:59.253451 2956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:31:59.262290 2956 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.3:46193
I20260619 14:31:59.262308 3084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.3:46193 every 8 connection(s)
I20260619 14:31:59.263171 2956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
I20260619 14:31:59.270982 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 2956
I20260619 14:31:59.271095 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879514993456-2056-0/minicluster-data/ts-2/wal/instance
I20260619 14:31:59.271395 2056 external_mini_cluster.cc:1428] Setting key e109466d5296ec879891492b95b555e5
Jun 19 14:31:59 dist-test-slave-70rm krb5kdc[2581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879519, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.3@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:31:59.276327 2665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:31:59.264887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.3:39943 (local address 127.2.2.62:46197)
0619 14:31:59.265018 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 14:31:59.265022 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:31:59.265824 (+ 802us) server_negotiation.cc:407] Connection header received
0619 14:31:59.267177 (+ 1353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:31:59.267180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:31:59.267229 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:31:59.267297 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:31:59.268459 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:31:59.269029 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:31:59.270106 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:31:59.270379 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:31:59.270458 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:31:59.273192 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:31:59.273195 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:31:59.273198 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:31:59.273227 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:31:59.274669 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:31:59.275120 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:31:59.275122 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:31:59.275123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:31:59.275192 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:31:59.275460 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:31:59.275462 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:31:59.275464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:31:59.275630 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:31:59.275701 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 14:31:59.276121 (+ 420us) server_negotiation.cc:299] Negotiation successful
0619 14:31:59.276210 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 14:31:59.277222 3087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:31:59.265265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46197 (local address 127.2.2.3:39943)
0619 14:31:59.265679 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0619 14:31:59.265720 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 14:31:59.266962 (+ 1242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:31:59.267456 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:31:59.267463 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:31:59.267825 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:31:59.268305 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:31:59.268318 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:31:59.269137 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:31:59.269140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:31:59.270002 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:31:59.270010 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:31:59.270466 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:31:59.270511 (+ 45us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:31:59.270663 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:31:59.271303 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:31:59.271326 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:31:59.273043 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:31:59.274774 (+ 1731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:31:59.274779 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:31:59.274791 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:31:59.275038 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:31:59.275262 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:31:59.275264 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:31:59.275265 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:31:59.275388 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:31:59.275706 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:31:59.275710 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:31:59.275931 (+ 221us) client_negotiation.cc:769] Sending connection context
0619 14:31:59.276104 (+ 173us) client_negotiation.cc:240] Negotiation successful
0619 14:31:59.276354 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":113,"threads_started":1}
I20260619 14:31:59.278357 3085 heartbeater.cc:347] Connected to a master server at 127.2.2.62:46197
I20260619 14:31:59.278625 3085 heartbeater.cc:464] Registering TS with master...
I20260619 14:31:59.279173 3085 heartbeater.cc:511] Master 127.2.2.62:46197 requested a full tablet report, sending...
I20260619 14:31:59.280185 2624 ts_manager.cc:194] Registered new tserver with Master: 80ec3f91128c49f09d7f0ef733624c65 (127.2.2.3:46193)
I20260619 14:31:59.280718 2624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.2.3@KRBTEST.COM'} at 127.2.2.3:39943
I20260619 14:31:59.285255 2056 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:31:59.295816 2665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:31:59.288475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41568 (local address 127.2.2.62:46197)
0619 14:31:59.288627 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 14:31:59.288631 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:31:59.288777 (+ 146us) server_negotiation.cc:407] Connection header received
0619 14:31:59.288933 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:31:59.288935 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:31:59.288974 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:31:59.289075 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:31:59.289745 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:31:59.290328 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:31:59.291139 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:31:59.291355 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:31:59.291456 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:31:59.292731 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:31:59.292734 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:31:59.292737 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:31:59.292762 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:31:59.294297 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:31:59.294738 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:31:59.294741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:31:59.294742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:31:59.294813 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:31:59.295040 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:31:59.295042 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:31:59.295043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:31:59.295249 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:31:59.295353 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 14:31:59.295611 (+ 258us) server_negotiation.cc:299] Negotiation successful
0619 14:31:59.295700 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 14:31:59.300746 2624 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41568:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:31:59.303797 2624 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:31:59.318001 3096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:31:59.311944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.2:36285 (local address 127.0.0.1:56962)
0619 14:31:59.313133 (+ 1189us) negotiation.cc:107] Waiting for socket to connect
0619 14:31:59.313147 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:31:59.313297 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:31:59.313825 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:31:59.313834 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:31:59.313858 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:31:59.314001 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:31:59.314011 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:31:59.315161 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:31:59.315166 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:31:59.316385 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:31:59.316394 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:31:59.317444 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:31:59.317448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:31:59.317552 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:31:59.317589 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 14:31:59.317706 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 14:31:59.317841 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"mutex_wait_us":2,"thread_start_us":98,"threads_started":1}
I20260619 14:31:59.318038 3098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:31:59.312108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56962 (local address 127.2.2.2:36285)
0619 14:31:59.313013 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0619 14:31:59.313025 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:31:59.313223 (+ 198us) server_negotiation.cc:407] Connection header received
0619 14:31:59.313472 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:31:59.313477 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:31:59.313668 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:31:59.313821 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:31:59.314138 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:31:59.315032 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:31:59.316498 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:31:59.317149 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:31:59.317247 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:31:59.317385 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 14:31:59.317757 (+ 372us) server_negotiation.cc:299] Negotiation successful
0619 14:31:59.317870 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":803,"thread_start_us":95,"threads_started":1}
I20260619 14:31:59.318758 3101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:31:59.312794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36280 (local address 127.2.2.3:46193)
0619 14:31:59.313318 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0619 14:31:59.313330 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:31:59.313365 (+ 35us) server_negotiation.cc:407] Connection header received
0619 14:31:59.313456 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:31:59.313461 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:31:59.313668 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:31:59.313843 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:31:59.314278 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:31:59.315455 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:31:59.316952 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:31:59.317737 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:31:59.317827 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:31:59.317945 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 14:31:59.318390 (+ 445us) server_negotiation.cc:299] Negotiation successful
0619 14:31:59.318619 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":138,"threads_started":1}
I20260619 14:31:59.318890 3097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:31:59.312020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.1:41169 (local address 127.0.0.1:44812)
0619 14:31:59.312857 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0619 14:31:59.312909 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 14:31:59.313033 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:31:59.314037 (+ 1004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:31:59.314043 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:31:59.314063 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:31:59.314182 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:31:59.314188 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:31:59.316005 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:31:59.316009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:31:59.317094 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:31:59.317100 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:31:59.318069 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:31:59.318073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:31:59.318562 (+ 489us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:31:59.318589 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 14:31:59.318679 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 14:31:59.318758 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"mutex_wait_us":50,"thread_start_us":113,"threads_started":1}
I20260619 14:31:59.319423 3099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:31:59.312387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44812 (local address 127.2.2.1:41169)
0619 14:31:59.313440 (+ 1053us) server_negotiation.cc:206] Beginning negotiation
0619 14:31:59.313455 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:31:59.313478 (+ 23us) server_negotiation.cc:407] Connection header received
0619 14:31:59.313559 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:31:59.313570 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:31:59.313741 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:31:59.313880 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:31:59.314320 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:31:59.315854 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:31:59.317196 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:31:59.317776 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:31:59.317875 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:31:59.318005 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 14:31:59.319200 (+ 1195us) server_negotiation.cc:299] Negotiation successful
0619 14:31:59.319301 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":943,"thread_start_us":89,"threads_started":1}
I20260619 14:31:59.321396 2885 tablet_service.cc:1467] Processing CreateTablet for tablet 4c37b854af5d4375a7d2cc4a96682f9c (DEFAULT_TABLE table=test-table [id=0f63693a2ef14d8aaf44ee00036f5e59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:31:59.322258 3100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:31:59.312640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.3:46193 (local address 127.0.0.1:36280)
0619 14:31:59.313186 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0619 14:31:59.313199 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:31:59.313273 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:31:59.313997 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:31:59.314000 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:31:59.314016 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:31:59.314162 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:31:59.314170 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:31:59.315596 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:31:59.315599 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:31:59.316824 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:31:59.316833 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:31:59.318001 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:31:59.318004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:31:59.318253 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:31:59.318282 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 14:31:59.318358 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 14:31:59.322094 (+ 3736us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"mutex_wait_us":121,"thread_start_us":95,"threads_started":1}
I20260619 14:31:59.321997 2751 tablet_service.cc:1467] Processing CreateTablet for tablet 4c37b854af5d4375a7d2cc4a96682f9c (DEFAULT_TABLE table=test-table [id=0f63693a2ef14d8aaf44ee00036f5e59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:31:59.322582 2885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c37b854af5d4375a7d2cc4a96682f9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:31:59.323047 2751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c37b854af5d4375a7d2cc4a96682f9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:31:59.324712 3019 tablet_service.cc:1467] Processing CreateTablet for tablet 4c37b854af5d4375a7d2cc4a96682f9c (DEFAULT_TABLE table=test-table [id=0f63693a2ef14d8aaf44ee00036f5e59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:31:59.325551 3019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c37b854af5d4375a7d2cc4a96682f9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:31:59.330049 3102 tablet_bootstrap.cc:492] T 4c37b854af5d4375a7d2cc4a96682f9c P a80837283a8645d49598f3fe3dbdb2b6: Bootstrap starting.
I20260619 14:31:59.330508 3103 tablet_bootstrap.cc:492] T 4c37b854af5d4375a7d2cc4a96682f9c P 0d5edcea117b436c9cc03ea3a0af7c09: Bootstrap starting.
I20260619 14:31:59.332770 3104 tablet_bootstrap.cc:492] T 4c37b854af5d4375a7d2cc4a96682f9c P 80ec3f91128c49f09d7f0ef733624c65: Bootstrap starting.
I20260619 14:31:59.333331 3103 tablet_bootstrap.cc:654] T 4c37b854af5d4375a7d2cc4a96682f9c P 0d5edcea117b436c9cc03ea3a0af7c09: Neither blocks nor log segments found. Creating new log.
I20260619 14:31:59.334324 3103 log.cc:824] T 4c37b854af5d4375a7d2cc4a96682f9c P 0d5edcea117b436c9cc03ea3a0af7c09: Log is configured to *not* fsync() on all Append() calls
I20260619 14:31:59.334990 3104 tablet_bootstrap.cc:654] T 4c37b854af5d4375a7d2cc4a96682f9c P 80ec3f91128c49f09d7f0ef733624c65: Neither blocks nor log segments found. Creating new log.
I20260619 14:31:59.334990 3102 tablet_bootstrap.cc:654] T 4c37b854af5d4375a7d2cc4a96682f9c P a80837283a8645d49598f3fe3dbdb2b6: Neither blocks nor log segments found. Creating new log.
I20260619 14:31:59.336325 3104 log.cc:824] T 4c37b854af5d4375a7d2cc4a96682f9c P 80ec3f91128c49f09d7f0ef733624c65: Log is configured to *not* fsync() on all Append() calls
I20260619 14:31:59.336326 3102 log.cc:824] T 4c37b854af5d4375a7d2cc4a96682f9c P a80837283a8645d49598f3fe3dbdb2b6: Log is configured to *not* fsync() on all Append() calls
I20260619 14:31:59.338550 3102 tablet_bootstrap.cc:492] T 4c37b854af5d4375a7d2cc4a96682f9c P a80837283a8645d49598f3fe3dbdb2b6: No bootstrap required, opened a new log
I20260619 14:31:59.338563 3103 tablet_bootstrap.cc:492] T 4c37b854af5d4375a7d2cc4a96682f9c P 0d5edcea117b436c9cc03ea3a0af7c09: No bootstrap required, opened a new log
I20260619 14:31:59.338568 3104 tablet_bootstrap.cc:492] T 4c37b854af5d4375a7d2cc4a96682f9c P 80ec3f91128c49f09d7f0ef733624c65: No bootstrap required, opened a new log
I20260619 14:31:59.338758 3102 ts_tablet_manager.cc:1397] T 4c37b854af5d4375a7d2cc4a96682f9c P a80837283a8645d49598f3fe3dbdb2b6: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.002s
I20260619 14:31:59.338760 3103 ts_tablet_manager.cc:1397] T 4c37b854af5d4375a7d2cc4a96682f9c P 0d5edcea117b436c9cc03ea3a0af7c09: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 14:31:59.338760 3104 ts_tablet_manager.cc:1397] T 4c37b854af5d4375a7d2cc4a96682f9c P 80ec3f91128c49f09d7f0ef733624c65: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 14:31:59.342402 3102 raft_consensus.cc:348] T 4c37b854af5d4375a7d2cc4a96682f9c P a80837283a8645d49598f3fe3dbdb2b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5edcea117b436c9cc03ea3a0af7c09" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 36285 } } peers { permanent_uuid: "a80837283a8645d49598f3fe3dbdb2b6" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 41169 } } peers { permanent_uuid: "80ec3f91128c49f09d7f0ef733624c65" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 46193 } }
I20260619 14:31:59.342675 3102 raft_consensus.cc:374] T 4c37b854af5d4375a7d2cc4a96682f9c P a80837283a8645d49598f3fe3dbdb2b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:31:59.342764 3102 raft_consensus.cc:729] T 4c37b854af5d4375a7d2cc4a96682f9c P a80837283a8645d49598f3fe3dbdb2b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a80837283a8645d49598f3fe3dbdb2b6, State: Initialized, Role: FOLLOWER
I20260619 14:31:59.342617 3103 raft_consensus.cc:348] T 4c37b854af5d4375a7d2cc4a96682f9c P 0d5edcea117b436c9cc03ea3a0af7c09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5edcea117b436c9cc03ea3a0af7c09" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 36285 } } peers { permanent_uuid: "a80837283a8645d49598f3fe3dbdb2b6" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 41169 } } peers { permanent_uuid: "80ec3f91128c49f09d7f0ef733624c65" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 46193 } }
I20260619 14:31:59.342622 3104 raft_consensus.cc:348] T 4c37b854af5d4375a7d2cc4a96682f9c P 80ec3f91128c49f09d7f0ef733624c65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5edcea117b436c9cc03ea3a0af7c09" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 36285 } } peers { permanent_uuid: "a80837283a8645d49598f3fe3dbdb2b6" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 41169 } } peers { permanent_uuid: "80ec3f91128c49f09d7f0ef733624c65" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 46193 } }
I20260619 14:31:59.342896 3104 raft_consensus.cc:374] T 4c37b854af5d4375a7d2cc4a96682f9c P 80ec3f91128c49f09d7f0ef733624c65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:31:59.342931 3103 raft_consensus.cc:374] T 4c37b854af5d4375a7d2cc4a96682f9c P 0d5edcea117b436c9cc03ea3a0af7c09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:31:59.342974 3104 raft_consensus.cc:729] T 4c37b854af5d4375a7d2cc4a96682f9c P 80ec3f91128c49f09d7f0ef733624c65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80ec3f91128c49f09d7f0ef733624c65, State: Initialized, Role: FOLLOWER
I20260619 14:31:59.342998 3103 raft_consensus.cc:729] T 4c37b854af5d4375a7d2cc4a96682f9c P 0d5edcea117b436c9cc03ea3a0af7c09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d5edcea117b436c9cc03ea3a0af7c09, State: Initialized, Role: FOLLOWER
I20260619 14:31:59.343313 3102 consensus_queue.cc:260] T 4c37b854af5d4375a7d2cc4a96682f9c P a80837283a8645d49598f3fe3dbdb2b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5edcea117b436c9cc03ea3a0af7c09" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 36285 } } peers { permanent_uuid: "a80837283a8645d49598f3fe3dbdb2b6" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 41169 } } peers { permanent_uuid: "80ec3f91128c49f09d7f0ef733624c65" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 46193 } }
I20260619 14:31:59.343492 3104 consensus_queue.cc:260] T 4c37b854af5d4375a7d2cc4a96682f9c P 80ec3f91128c49f09d7f0ef733624c65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5edcea117b436c9cc03ea3a0af7c09" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 36285 } } peers { permanent_uuid: "a80837283a8645d49598f3fe3dbdb2b6" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 41169 } } peers { permanent_uuid: "80ec3f91128c49f09d7f0ef733624c65" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 46193 } }
I20260619 14:31:59.343492 3103 consensus_queue.cc:260] T 4c37b854af5d4375a7d2cc4a96682f9c P 0d5edcea117b436c9cc03ea3a0af7c09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5edcea117b436c9cc03ea3a0af7c09" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 36285 } } peers { permanent_uuid: "a80837283a8645d49598f3fe3dbdb2b6" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 41169 } } peers { permanent_uuid: "80ec3f91128c49f09d7f0ef733624c65" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 46193 } }
I20260619 14:31:59.344064 2817 heartbeater.cc:503] Master 127.2.2.62:46197 was elected leader, sending a full tablet report...
I20260619 14:31:59.344403 3102 ts_tablet_manager.cc:1428] T 4c37b854af5d4375a7d2cc4a96682f9c P a80837283a8645d49598f3fe3dbdb2b6: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260619 14:31:59.344441 3104 ts_tablet_manager.cc:1428] T 4c37b854af5d4375a7d2cc4a96682f9c P 80ec3f91128c49f09d7f0ef733624c65: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 14:31:59.344441 3103 ts_tablet_manager.cc:1428] T 4c37b854af5d4375a7d2cc4a96682f9c P 0d5edcea117b436c9cc03ea3a0af7c09: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 14:31:59.344774 2951 heartbeater.cc:503] Master 127.2.2.62:46197 was elected leader, sending a full tablet report...
I20260619 14:31:59.345237 3085 heartbeater.cc:503] Master 127.2.2.62:46197 was elected leader, sending a full tablet report...
W20260619 14:31:59.514092 3086 tablet.cc:2369] T 4c37b854af5d4375a7d2cc4a96682f9c P 80ec3f91128c49f09d7f0ef733624c65: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:31:59.548369 2952 tablet.cc:2369] T 4c37b854af5d4375a7d2cc4a96682f9c P 0d5edcea117b436c9cc03ea3a0af7c09: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:31:59.587095 2818 tablet.cc:2369] T 4c37b854af5d4375a7d2cc4a96682f9c P a80837283a8645d49598f3fe3dbdb2b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:31:59.708432 3108 raft_consensus.cc:482] T 4c37b854af5d4375a7d2cc4a96682f9c P a80837283a8645d49598f3fe3dbdb2b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:31:59.708695 3108 raft_consensus.cc:504] T 4c37b854af5d4375a7d2cc4a96682f9c P a80837283a8645d49598f3fe3dbdb2b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5edcea117b436c9cc03ea3a0af7c09" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 36285 } } peers { permanent_uuid: "a80837283a8645d49598f3fe3dbdb2b6" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 41169 } } peers { permanent_uuid: "80ec3f91128c49f09d7f0ef733624c65" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 46193 } }
I20260619 14:31:59.709841 3108 leader_election.cc:288] T 4c37b854af5d4375a7d2cc4a96682f9c P a80837283a8645d49598f3fe3dbdb2b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d5edcea117b436c9cc03ea3a0af7c09 (127.2.2.2:36285), 80ec3f91128c49f09d7f0ef733624c65 (127.2.2.3:46193)
I20260619 14:31:59.713781 3098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:31:59.710177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:58623 (local address 127.2.2.2:36285)
0619 14:31:59.710322 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 14:31:59.710327 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:31:59.710534 (+ 207us) server_negotiation.cc:407] Connection header received
0619 14:31:59.710692 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:31:59.710698 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:31:59.710760 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:31:59.710823 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:31:59.711127 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:31:59.711639 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:31:59.712790 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:31:59.713207 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:31:59.713322 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:31:59.713383 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 14:31:59.713565 (+ 182us) server_negotiation.cc:299] Negotiation successful
0619 14:31:59.713643 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 14:31:59.713799 3112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:31:59.710052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.2:36285 (local address 127.2.2.1:58623)
0619 14:31:59.710432 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0619 14:31:59.710452 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 14:31:59.710551 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:31:59.710851 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:31:59.710853 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:31:59.710874 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:31:59.711024 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:31:59.711029 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:31:59.711742 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:31:59.711746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:31:59.712675 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:31:59.712684 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:31:59.713318 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:31:59.713321 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:31:59.713467 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:31:59.713486 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 14:31:59.713544 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 14:31:59.713635 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":49,"thread_start_us":120,"threads_started":1}
I20260619 14:31:59.713801 3101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:31:59.710288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:40377 (local address 127.2.2.3:46193)
0619 14:31:59.710407 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 14:31:59.710412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:31:59.710626 (+ 214us) server_negotiation.cc:407] Connection header received
0619 14:31:59.710777 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:31:59.710779 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:31:59.710842 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:31:59.710911 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:31:59.711184 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:31:59.711648 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:31:59.712793 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:31:59.713212 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:31:59.713322 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:31:59.713384 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 14:31:59.713633 (+ 249us) server_negotiation.cc:299] Negotiation successful
0619 14:31:59.713703 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 14:31:59.713852 3111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:31:59.710173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.3:46193 (local address 127.2.2.1:40377)
0619 14:31:59.710519 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0619 14:31:59.710534 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:31:59.710661 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:31:59.710932 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:31:59.710935 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:31:59.710956 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:31:59.711064 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:31:59.711069 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:31:59.711766 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:31:59.711769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:31:59.712678 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:31:59.712687 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:31:59.713336 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:31:59.713340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:31:59.713502 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:31:59.713523 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 14:31:59.713602 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 14:31:59.713666 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":47,"thread_start_us":96,"threads_started":1}
I20260619 14:31:59.714457 3109 raft_consensus.cc:482] T 4c37b854af5d4375a7d2cc4a96682f9c P 0d5edcea117b436c9cc03ea3a0af7c09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:31:59.714702 3039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c37b854af5d4375a7d2cc4a96682f9c" candidate_uuid: "a80837283a8645d49598f3fe3dbdb2b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80ec3f91128c49f09d7f0ef733624c65" is_pre_election: true
I20260619 14:31:59.715018 3109 raft_consensus.cc:504] T 4c37b854af5d4375a7d2cc4a96682f9c P 0d5edcea117b436c9cc03ea3a0af7c09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5edcea117b436c9cc03ea3a0af7c09" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 36285 } } peers { permanent_uuid: "a80837283a8645d49598f3fe3dbdb2b6" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 41169 } } peers { permanent_uuid: "80ec3f91128c49f09d7f0ef733624c65" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 46193 } }
I20260619 14:31:59.715118 2905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c37b854af5d4375a7d2cc4a96682f9c" candidate_uuid: "a80837283a8645d49598f3fe3dbdb2b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d5edcea117b436c9cc03ea3a0af7c09" is_pre_election: true
I20260619 14:31:59.715169 3039 raft_consensus.cc:2454] T 4c37b854af5d4375a7d2cc4a96682f9c P 80ec3f91128c49f09d7f0ef733624c65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80837283a8645d49598f3fe3dbdb2b6 in term 0.
I20260619 14:31:59.715462 2905 raft_consensus.cc:2454] T 4c37b854af5d4375a7d2cc4a96682f9c P 0d5edcea117b436c9cc03ea3a0af7c09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80837283a8645d49598f3fe3dbdb2b6 in term 0.
I20260619 14:31:59.715669 2705 leader_election.cc:302] T 4c37b854af5d4375a7d2cc4a96682f9c P a80837283a8645d49598f3fe3dbdb2b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80ec3f91128c49f09d7f0ef733624c65, a80837283a8645d49598f3fe3dbdb2b6; no voters:
I20260619 14:31:59.715998 3108 raft_consensus.cc:2781] T 4c37b854af5d4375a7d2cc4a96682f9c P a80837283a8645d49598f3fe3dbdb2b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:31:59.716082 3108 raft_consensus.cc:482] T 4c37b854af5d4375a7d2cc4a96682f9c P a80837283a8645d49598f3fe3dbdb2b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:31:59.716125 3108 raft_consensus.cc:3037] T 4c37b854af5d4375a7d2cc4a96682f9c P a80837283a8645d49598f3fe3dbdb2b6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:31:59.716287 3109 leader_election.cc:288] T 4c37b854af5d4375a7d2cc4a96682f9c P 0d5edcea117b436c9cc03ea3a0af7c09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a80837283a8645d49598f3fe3dbdb2b6 (127.2.2.1:41169), 80ec3f91128c49f09d7f0ef733624c65 (127.2.2.3:46193)
I20260619 14:31:59.717456 3108 raft_consensus.cc:504] T 4c37b854af5d4375a7d2cc4a96682f9c P a80837283a8645d49598f3fe3dbdb2b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5edcea117b436c9cc03ea3a0af7c09" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 36285 } } peers { permanent_uuid: "a80837283a8645d49598f3fe3dbdb2b6" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 41169 } } peers { permanent_uuid: "80ec3f91128c49f09d7f0ef733624c65" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 46193 } }
I20260619 14:31:59.718000 3108 leader_election.cc:288] T 4c37b854af5d4375a7d2cc4a96682f9c P a80837283a8645d49598f3fe3dbdb2b6 [CANDIDATE]: Term 1 election: Requested vote from peers 0d5edcea117b436c9cc03ea3a0af7c09 (127.2.2.2:36285), 80ec3f91128c49f09d7f0ef733624c65 (127.2.2.3:46193)
I20260619 14:31:59.718199 2905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c37b854af5d4375a7d2cc4a96682f9c" candidate_uuid: "a80837283a8645d49598f3fe3dbdb2b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d5edcea117b436c9cc03ea3a0af7c09"
I20260619 14:31:59.718328 2905 raft_consensus.cc:3037] T 4c37b854af5d4375a7d2cc4a96682f9c P 0d5edcea117b436c9cc03ea3a0af7c09 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:31:59.719314 2905 raft_consensus.cc:2454] T 4c37b854af5d4375a7d2cc4a96682f9c P 0d5edcea117b436c9cc03ea3a0af7c09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a80837283a8645d49598f3fe3dbdb2b6 in term 1.
I20260619 14:31:59.719643 2704 leader_election.cc:302] T 4c37b854af5d4375a7d2cc4a96682f9c P a80837283a8645d49598f3fe3dbdb2b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d5edcea117b436c9cc03ea3a0af7c09, a80837283a8645d49598f3fe3dbdb2b6; no voters:
I20260619 14:31:59.719674 3039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c37b854af5d4375a7d2cc4a96682f9c" candidate_uuid: "a80837283a8645d49598f3fe3dbdb2b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80ec3f91128c49f09d7f0ef733624c65"
I20260619 14:31:59.719779 3039 raft_consensus.cc:3037] T 4c37b854af5d4375a7d2cc4a96682f9c P 80ec3f91128c49f09d7f0ef733624c65 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:31:59.720094 3108 raft_consensus.cc:2781] T 4c37b854af5d4375a7d2cc4a96682f9c P a80837283a8645d49598f3fe3dbdb2b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:31:59.720345 3108 raft_consensus.cc:686] T 4c37b854af5d4375a7d2cc4a96682f9c P a80837283a8645d49598f3fe3dbdb2b6 [term 1 LEADER]: Becoming Leader. State: Replica: a80837283a8645d49598f3fe3dbdb2b6, State: Running, Role: LEADER
I20260619 14:31:59.720736 3108 consensus_queue.cc:237] T 4c37b854af5d4375a7d2cc4a96682f9c P a80837283a8645d49598f3fe3dbdb2b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5edcea117b436c9cc03ea3a0af7c09" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 36285 } } peers { permanent_uuid: "a80837283a8645d49598f3fe3dbdb2b6" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 41169 } } peers { permanent_uuid: "80ec3f91128c49f09d7f0ef733624c65" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 46193 } }
I20260619 14:31:59.720851 3113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:31:59.716428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.1:41169 (local address 127.2.2.2:58135)
0619 14:31:59.716646 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0619 14:31:59.716662 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:31:59.716783 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:31:59.717129 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:31:59.717131 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:31:59.717145 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:31:59.717218 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:31:59.717222 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:31:59.718104 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:31:59.718107 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:31:59.719357 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:31:59.719367 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:31:59.720451 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:31:59.720455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:31:59.720544 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:31:59.720566 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 14:31:59.720635 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 14:31:59.720699 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":64,"threads_started":1}
I20260619 14:31:59.721081 3039 raft_consensus.cc:2454] T 4c37b854af5d4375a7d2cc4a96682f9c P 80ec3f91128c49f09d7f0ef733624c65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a80837283a8645d49598f3fe3dbdb2b6 in term 1.
I20260619 14:31:59.721256 3099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:31:59.716577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:58135 (local address 127.2.2.1:41169)
0619 14:31:59.716666 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0619 14:31:59.716670 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:31:59.716720 (+ 50us) server_negotiation.cc:407] Connection header received
0619 14:31:59.716900 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:31:59.716903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:31:59.716948 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:31:59.716993 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:31:59.717320 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:31:59.717986 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:31:59.719616 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:31:59.720211 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:31:59.720317 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:31:59.720390 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 14:31:59.721091 (+ 701us) server_negotiation.cc:299] Negotiation successful
0619 14:31:59.721155 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 14:31:59.721382 3101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:31:59.716833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:38473 (local address 127.2.2.3:46193)
0619 14:31:59.717008 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 14:31:59.717012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:31:59.717517 (+ 505us) server_negotiation.cc:407] Connection header received
0619 14:31:59.717613 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:31:59.717616 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:31:59.717663 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:31:59.717734 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:31:59.717980 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:31:59.718720 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:31:59.720050 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:31:59.720697 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:31:59.720797 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:31:59.720848 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 14:31:59.721195 (+ 347us) server_negotiation.cc:299] Negotiation successful
0619 14:31:59.721266 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":95}
I20260619 14:31:59.721685 2771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c37b854af5d4375a7d2cc4a96682f9c" candidate_uuid: "0d5edcea117b436c9cc03ea3a0af7c09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80837283a8645d49598f3fe3dbdb2b6" is_pre_election: true
I20260619 14:31:59.721801 3114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:31:59.716750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.3:46193 (local address 127.2.2.2:38473)
0619 14:31:59.717423 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0619 14:31:59.717435 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:31:59.717522 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:31:59.717752 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:31:59.717755 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:31:59.717770 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:31:59.717864 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:31:59.717869 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:31:59.718848 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:31:59.718851 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:31:59.719936 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:31:59.719945 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:31:59.720910 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:31:59.720913 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:31:59.721046 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:31:59.721073 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 14:31:59.721594 (+ 521us) client_negotiation.cc:240] Negotiation successful
0619 14:31:59.721671 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":77,"threads_started":1}
I20260619 14:31:59.722918 3039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c37b854af5d4375a7d2cc4a96682f9c" candidate_uuid: "0d5edcea117b436c9cc03ea3a0af7c09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80ec3f91128c49f09d7f0ef733624c65" is_pre_election: true
I20260619 14:31:59.723085 3039 raft_consensus.cc:2379] T 4c37b854af5d4375a7d2cc4a96682f9c P 80ec3f91128c49f09d7f0ef733624c65 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d5edcea117b436c9cc03ea3a0af7c09 in current term 1: Already voted for candidate a80837283a8645d49598f3fe3dbdb2b6 in this term.
I20260619 14:31:59.723531 2839 leader_election.cc:302] T 4c37b854af5d4375a7d2cc4a96682f9c P 0d5edcea117b436c9cc03ea3a0af7c09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d5edcea117b436c9cc03ea3a0af7c09; no voters: 80ec3f91128c49f09d7f0ef733624c65, a80837283a8645d49598f3fe3dbdb2b6
I20260619 14:31:59.723773 3109 raft_consensus.cc:2726] T 4c37b854af5d4375a7d2cc4a96682f9c P 0d5edcea117b436c9cc03ea3a0af7c09 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260619 14:31:59.724153 2624 catalog_manager.cc:5495] T 4c37b854af5d4375a7d2cc4a96682f9c P a80837283a8645d49598f3fe3dbdb2b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to a80837283a8645d49598f3fe3dbdb2b6 (127.2.2.1). New cstate: current_term: 1 leader_uuid: "a80837283a8645d49598f3fe3dbdb2b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5edcea117b436c9cc03ea3a0af7c09" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 36285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a80837283a8645d49598f3fe3dbdb2b6" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 41169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80ec3f91128c49f09d7f0ef733624c65" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 46193 } health_report { overall_health: UNKNOWN } } }
Jun 19 14:31:59 dist-test-slave-70rm krb5kdc[2581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.2.1@KRBTEST.COM
I20260619 14:31:59.770347 3099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:31:59.761968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44814 (local address 127.2.2.1:41169)
0619 14:31:59.762172 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 14:31:59.762177 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:31:59.762322 (+ 145us) server_negotiation.cc:407] Connection header received
0619 14:31:59.762427 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:31:59.762429 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:31:59.762487 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:31:59.762557 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:31:59.763204 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:31:59.763711 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:31:59.764495 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:31:59.764701 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:31:59.764776 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:31:59.767233 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:31:59.767244 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:31:59.767258 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:31:59.767282 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:31:59.768800 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:31:59.769177 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:31:59.769182 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:31:59.769183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:31:59.769256 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:31:59.769463 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:31:59.769465 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:31:59.769467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:31:59.769701 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:31:59.769866 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 14:31:59.769981 (+ 115us) server_negotiation.cc:299] Negotiation successful
0619 14:31:59.770133 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
Jun 19 14:31:59 dist-test-slave-70rm krb5kdc[2581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879519, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 14:31:59 dist-test-slave-70rm krb5kdc[2581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879519, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.2.1@KRBTEST.COM
I20260619 14:31:59.797464 3099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:31:59.788103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44826 (local address 127.2.2.1:41169)
0619 14:31:59.788257 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 14:31:59.788261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:31:59.788698 (+ 437us) server_negotiation.cc:407] Connection header received
0619 14:31:59.788825 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:31:59.788828 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:31:59.788872 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:31:59.788944 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:31:59.789662 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:31:59.790259 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:31:59.791256 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:31:59.791500 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:31:59.791629 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:31:59.794226 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:31:59.794229 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:31:59.794231 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:31:59.794257 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:31:59.795753 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:31:59.796221 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:31:59.796224 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:31:59.796225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:31:59.796302 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:31:59.796565 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:31:59.796569 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:31:59.796571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:31:59.796752 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:31:59.796909 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 14:31:59.797173 (+ 264us) server_negotiation.cc:299] Negotiation successful
0619 14:31:59.797297 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
Jun 19 14:31:59 dist-test-slave-70rm krb5kdc[2581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879519, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 14:31:59 dist-test-slave-70rm krb5kdc[2581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879519, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.2.1@KRBTEST.COM
I20260619 14:31:59.824512 3099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:31:59.815527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44834 (local address 127.2.2.1:41169)
0619 14:31:59.815689 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 14:31:59.815694 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:31:59.815837 (+ 143us) server_negotiation.cc:407] Connection header received
0619 14:31:59.816036 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:31:59.816039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:31:59.816090 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:31:59.816177 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:31:59.816889 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:31:59.817393 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:31:59.818316 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:31:59.818543 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:31:59.818677 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:31:59.821006 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:31:59.821013 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:31:59.821018 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:31:59.821046 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:31:59.822544 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:31:59.823016 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:31:59.823020 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:31:59.823021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:31:59.823109 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:31:59.823404 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:31:59.823407 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:31:59.823409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:31:59.823586 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:31:59.824195 (+ 609us) server_negotiation.cc:1035] Waiting for connection context
0619 14:31:59.824274 (+ 79us) server_negotiation.cc:299] Negotiation successful
0619 14:31:59.824385 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260619 14:31:59.825829 2731 tablet_service.cc:2908] Rejecting scan request for tablet 4c37b854af5d4375a7d2cc4a96682f9c: Uninitialized: clean time has not yet been initialized
I20260619 14:31:59.827875 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 2688
I20260619 14:31:59.835508 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 2822
I20260619 14:31:59.841151 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 2956
I20260619 14:31:59.846163 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 2594
2026-06-19T14:31:59Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2626 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:31:59 dist-test-slave-70rm krb5kdc[3129](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:31:59 dist-test-slave-70rm krb5kdc[3129](info): set up 2 sockets
Jun 19 14:31:59 dist-test-slave-70rm krb5kdc[3129](info): commencing operation
krb5kdc: starting...
W20260619 14:32:00.885929 2056 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Jun 19 14:32:00 dist-test-slave-70rm krb5kdc[3129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:32:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:32:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.2.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab.
Entry for principal kudu/127.2.2.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab.
I20260619 14:32:01.015560 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:34767
--webserver_interface=127.2.2.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:33365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.2.62:34767
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:32:01.129465 3142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:01.129755 3142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:01.129837 3142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:01.134228 3142 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:32:01.134284 3142 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 14:32:01.134302 3142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:01.134320 3142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:01.134367 3142 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:32:01.134394 3142 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:32:01.139648 3142 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:33365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.2.62:34767
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:34767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.2.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:32:01.140694 3142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:01.141774 3142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:32:01.150056 3147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:01.150173 3142 server_base.cc:1034] running on GCE node
W20260619 14:32:01.150056 3148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:01.150029 3150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:01.150902 3142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:01.152024 3142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:01.153203 3142 hybrid_clock.cc:648] HybridClock initialized: now 1781879521153191 us; error 29 us; skew 500 ppm
Jun 19 14:32:01 dist-test-slave-70rm krb5kdc[3129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879521, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:32:01.155994 3142 init.cc:377] Logged in from keytab as kudu/127.2.2.62@KRBTEST.COM (short username kudu)
I20260619 14:32:01.156981 3142 webserver.cc:460] Webserver started at http://127.2.2.62:37967/ using document root <none> and password file <none>
I20260619 14:32:01.157580 3142 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:01.157677 3142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:01.157913 3142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:32:01.160117 3142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/master-0/data/instance:
uuid: "25aefb30399c43b7a1516ee1783654fb"
format_stamp: "Formatted at 2026-06-19 14:32:01 on dist-test-slave-70rm"
server_key: "a6069f82d124b864763d62114b2cd65d"
server_key_iv: "4c1daa39ae22b9851e9c620f0af40135"
server_key_version: "encryptionkey@0"
I20260619 14:32:01.160763 3142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/master-0/wal/instance:
uuid: "25aefb30399c43b7a1516ee1783654fb"
format_stamp: "Formatted at 2026-06-19 14:32:01 on dist-test-slave-70rm"
server_key: "a6069f82d124b864763d62114b2cd65d"
server_key_iv: "4c1daa39ae22b9851e9c620f0af40135"
server_key_version: "encryptionkey@0"
I20260619 14:32:01.164783 3142 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 14:32:01.167541 3157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:01.169024 3142 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 14:32:01.169167 3142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/master-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/master-0/wal
uuid: "25aefb30399c43b7a1516ee1783654fb"
format_stamp: "Formatted at 2026-06-19 14:32:01 on dist-test-slave-70rm"
server_key: "a6069f82d124b864763d62114b2cd65d"
server_key_iv: "4c1daa39ae22b9851e9c620f0af40135"
server_key_version: "encryptionkey@0"
I20260619 14:32:01.169286 3142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:01.202284 3142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:01.205615 3142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:32:01.205816 3142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:01.213457 3142 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.62:34767
I20260619 14:32:01.213490 3219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.62:34767 every 8 connection(s)
I20260619 14:32:01.214425 3142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
I20260619 14:32:01.217193 3220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:01.222608 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 3142
I20260619 14:32:01.222738 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/master-0/wal/instance
I20260619 14:32:01.222983 2056 external_mini_cluster.cc:1428] Setting key 8c2cb5a8fb0e924e5c17483b6106fc77
I20260619 14:32:01.223140 3220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25aefb30399c43b7a1516ee1783654fb: Bootstrap starting.
I20260619 14:32:01.225669 3220 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25aefb30399c43b7a1516ee1783654fb: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:01.226586 3220 log.cc:824] T 00000000000000000000000000000000 P 25aefb30399c43b7a1516ee1783654fb: Log is configured to *not* fsync() on all Append() calls
I20260619 14:32:01.228735 3220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25aefb30399c43b7a1516ee1783654fb: No bootstrap required, opened a new log
Jun 19 14:32:01 dist-test-slave-70rm krb5kdc[3129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:32:01.233182 3220 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25aefb30399c43b7a1516ee1783654fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25aefb30399c43b7a1516ee1783654fb" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 34767 } }
I20260619 14:32:01.233467 3220 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25aefb30399c43b7a1516ee1783654fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:01.233543 3220 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25aefb30399c43b7a1516ee1783654fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25aefb30399c43b7a1516ee1783654fb, State: Initialized, Role: FOLLOWER
I20260619 14:32:01.234046 3220 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25aefb30399c43b7a1516ee1783654fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25aefb30399c43b7a1516ee1783654fb" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 34767 } }
I20260619 14:32:01.234223 3220 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25aefb30399c43b7a1516ee1783654fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:32:01.234313 3220 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25aefb30399c43b7a1516ee1783654fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:32:01.234413 3220 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25aefb30399c43b7a1516ee1783654fb [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:01.235343 3220 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25aefb30399c43b7a1516ee1783654fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25aefb30399c43b7a1516ee1783654fb" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 34767 } }
I20260619 14:32:01.235747 3220 leader_election.cc:302] T 00000000000000000000000000000000 P 25aefb30399c43b7a1516ee1783654fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25aefb30399c43b7a1516ee1783654fb; no voters:
I20260619 14:32:01.236096 3220 leader_election.cc:288] T 00000000000000000000000000000000 P 25aefb30399c43b7a1516ee1783654fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:32:01.236192 3225 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25aefb30399c43b7a1516ee1783654fb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:32:01.236361 3225 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25aefb30399c43b7a1516ee1783654fb [term 1 LEADER]: Becoming Leader. State: Replica: 25aefb30399c43b7a1516ee1783654fb, State: Running, Role: LEADER
I20260619 14:32:01.236660 3225 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25aefb30399c43b7a1516ee1783654fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25aefb30399c43b7a1516ee1783654fb" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 34767 } }
I20260619 14:32:01.237406 3220 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25aefb30399c43b7a1516ee1783654fb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:32:01.238430 3227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25aefb30399c43b7a1516ee1783654fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25aefb30399c43b7a1516ee1783654fb. Latest consensus state: current_term: 1 leader_uuid: "25aefb30399c43b7a1516ee1783654fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25aefb30399c43b7a1516ee1783654fb" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 34767 } } }
I20260619 14:32:01.238482 3223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:01.224406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33032 (local address 127.2.2.62:34767)
0619 14:32:01.224820 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:01.224837 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:01.224870 (+ 33us) server_negotiation.cc:407] Connection header received
0619 14:32:01.225497 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:01.225514 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:01.225904 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:01.226257 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:01.227138 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:01.228216 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:01.229329 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:01.229802 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:01.229930 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:01.232734 (+ 2804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:01.232739 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:01.232756 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:01.232784 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:01.234821 (+ 2037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:01.235269 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:01.235275 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:01.235282 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:01.235400 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:01.235680 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:01.235683 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:01.235685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:01.236140 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:01.237042 (+ 902us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:01.237172 (+ 130us) server_negotiation.cc:299] Negotiation successful
0619 14:32:01.237406 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":139,"threads_started":1}
I20260619 14:32:01.238564 3227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25aefb30399c43b7a1516ee1783654fb [sys.catalog]: This master's current role is: LEADER
I20260619 14:32:01.238485 3226 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25aefb30399c43b7a1516ee1783654fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25aefb30399c43b7a1516ee1783654fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25aefb30399c43b7a1516ee1783654fb" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 34767 } } }
I20260619 14:32:01.238629 3226 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25aefb30399c43b7a1516ee1783654fb [sys.catalog]: This master's current role is: LEADER
W20260619 14:32:01.242033 3240 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 25aefb30399c43b7a1516ee1783654fb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:32:01.242100 3240 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:32:01.242218 3234 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:32:01.243135 3234 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:32:01.248785 3234 catalog_manager.cc:1347] Generated new cluster ID: e16748ff33b4425183db3fcb4f23a3f1
I20260619 14:32:01.248867 3234 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:32:01.257876 3234 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:32:01.259207 3234 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:32:01.270048 3234 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25aefb30399c43b7a1516ee1783654fb: Generated new TSK 0
I20260619 14:32:01.270787 3234 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.2.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab.
Entry for principal kudu/127.2.2.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab.
I20260619 14:32:01.305666 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.1:0
--local_ip_for_outbound_sockets=127.2.2.1
--webserver_interface=127.2.2.1
--webserver_port=0
--tserver_master_addrs=127.2.2.62:34767
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab
--principal=kudu/127.2.2.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:33365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:32:01.430720 3246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:01.431020 3246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:01.431067 3246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:01.435271 3246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:01.435353 3246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:01.435395 3246 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 14:32:01.435431 3246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.1
I20260619 14:32:01.441398 3246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:33365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab
--principal=kudu/127.2.2.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.2.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.2.62:34767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3246
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.1
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:32:01.442615 3246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:01.443697 3246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:32:01.453740 3254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:01.453735 3251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:01.453974 3246 server_base.cc:1034] running on GCE node
W20260619 14:32:01.453740 3252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:01.454591 3246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:01.455340 3246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:01.456562 3246 hybrid_clock.cc:648] HybridClock initialized: now 1781879521456521 us; error 60 us; skew 500 ppm
Jun 19 14:32:01 dist-test-slave-70rm krb5kdc[3129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879521, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:32:01.464248 3246 init.cc:377] Logged in from keytab as kudu/127.2.2.1@KRBTEST.COM (short username kudu)
I20260619 14:32:01.465296 3246 webserver.cc:460] Webserver started at http://127.2.2.1:35825/ using document root <none> and password file <none>
I20260619 14:32:01.465947 3246 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:01.466053 3246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:01.466361 3246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:32:01.468555 3246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/ts-0/data/instance:
uuid: "fa0316c331a843ba89f6c3258d151bca"
format_stamp: "Formatted at 2026-06-19 14:32:01 on dist-test-slave-70rm"
server_key: "3e0d42c64bba0b30370618e0bfccc333"
server_key_iv: "fad5a7fcd510e558fba623427775ba30"
server_key_version: "encryptionkey@0"
I20260619 14:32:01.469172 3246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/ts-0/wal/instance:
uuid: "fa0316c331a843ba89f6c3258d151bca"
format_stamp: "Formatted at 2026-06-19 14:32:01 on dist-test-slave-70rm"
server_key: "3e0d42c64bba0b30370618e0bfccc333"
server_key_iv: "fad5a7fcd510e558fba623427775ba30"
server_key_version: "encryptionkey@0"
I20260619 14:32:01.473384 3246 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 14:32:01.476404 3261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:01.477859 3246 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:32:01.478075 3246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/ts-0/wal
uuid: "fa0316c331a843ba89f6c3258d151bca"
format_stamp: "Formatted at 2026-06-19 14:32:01 on dist-test-slave-70rm"
server_key: "3e0d42c64bba0b30370618e0bfccc333"
server_key_iv: "fad5a7fcd510e558fba623427775ba30"
server_key_version: "encryptionkey@0"
I20260619 14:32:01.478283 3246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:01.498576 3246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:01.502522 3246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:32:01.502743 3246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:01.504259 3246 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:32:01.504340 3246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:01.504410 3246 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:32:01.504454 3246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 14:32:01 dist-test-slave-70rm krb5kdc[3129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879521, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.1@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:32:01.515774 3246 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.1:35249
I20260619 14:32:01.515931 3376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.1:35249 every 8 connection(s)
I20260619 14:32:01.516937 3246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
I20260619 14:32:01.519513 3223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:01.505307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:48177 (local address 127.2.2.62:34767)
0619 14:32:01.505457 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:01.505462 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:01.506469 (+ 1007us) server_negotiation.cc:407] Connection header received
0619 14:32:01.507570 (+ 1101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:01.507575 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:01.507658 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:01.507769 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:01.509537 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:01.510482 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:01.511897 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:01.512158 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:01.512278 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:01.515271 (+ 2993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:01.515275 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:01.515277 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:01.515307 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:01.516995 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:01.517745 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:01.517749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:01.517754 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:01.517861 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:01.518319 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:01.518322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:01.518324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:01.518533 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:01.518655 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:01.519185 (+ 530us) server_negotiation.cc:299] Negotiation successful
0619 14:32:01.519333 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 14:32:01.520373 3270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:01.505781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:34767 (local address 127.2.2.1:48177)
0619 14:32:01.506295 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:01.506348 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:01.507332 (+ 984us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:01.508011 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:01.508023 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:01.508471 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:01.509348 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:01.509376 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:01.510647 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:01.510651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:01.511757 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:01.511770 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:01.512262 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:01.512327 (+ 65us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:01.512523 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:01.513312 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:01.513331 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:01.515043 (+ 1712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:01.517161 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:01.517168 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:01.517185 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:01.517594 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:01.517980 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:01.517983 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:01.517985 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:01.518214 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:01.518652 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:01.518659 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:01.518935 (+ 276us) client_negotiation.cc:769] Sending connection context
0619 14:32:01.519130 (+ 195us) client_negotiation.cc:240] Negotiation successful
0619 14:32:01.519479 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":136,"threads_started":1}
I20260619 14:32:01.521847 3377 heartbeater.cc:347] Connected to a master server at 127.2.2.62:34767
I20260619 14:32:01.522027 3377 heartbeater.cc:464] Registering TS with master...
I20260619 14:32:01.522590 3377 heartbeater.cc:511] Master 127.2.2.62:34767 requested a full tablet report, sending...
I20260619 14:32:01.523180 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 3246
I20260619 14:32:01.523298 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/ts-0/wal/instance
I20260619 14:32:01.523622 2056 external_mini_cluster.cc:1428] Setting key 142768ec6190211a1d2c32ca95e6e919
I20260619 14:32:01.524708 3173 ts_manager.cc:194] Registered new tserver with Master: fa0316c331a843ba89f6c3258d151bca (127.2.2.1:35249)
I20260619 14:32:01.526006 3173 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.2.1@KRBTEST.COM'} at 127.2.2.1:48177
I20260619 14:32:01.532426 3223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:01.524443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:35387 (local address 127.2.2.62:34767)
0619 14:32:01.524575 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:01.524579 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:01.524913 (+ 334us) server_negotiation.cc:407] Connection header received
0619 14:32:01.525059 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:01.525062 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:01.525109 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:01.525179 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:01.525889 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:01.526641 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:01.527616 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:01.527838 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:01.527947 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:01.529179 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:01.529183 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:01.529186 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:01.529216 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:01.530727 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:01.531189 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:01.531193 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:01.531196 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:01.531280 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:01.531552 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:01.531557 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:01.531559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:01.531751 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:01.531854 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:01.532125 (+ 271us) server_negotiation.cc:299] Negotiation successful
0619 14:32:01.532239 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 14:32:01.532442 3383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:01.524401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:34767 (local address 127.2.2.1:35387)
0619 14:32:01.524835 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:01.524853 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:01.524958 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:01.525298 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:01.525301 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:01.525542 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:01.525744 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:01.525757 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:01.526806 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:01.526811 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:01.527486 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:01.527496 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:01.527935 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:01.527957 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:01.528078 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:01.528641 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:01.528653 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:01.529023 (+ 370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:01.530852 (+ 1829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:01.530856 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:01.530858 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:01.531081 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:01.531371 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:01.531375 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:01.531377 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:01.531463 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:01.531852 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:01.531855 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:01.531955 (+ 100us) client_negotiation.cc:769] Sending connection context
0619 14:32:01.532070 (+ 115us) client_negotiation.cc:240] Negotiation successful
0619 14:32:01.532237 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":114,"threads_started":1}
WARNING: no policy specified for kudu/127.2.2.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab.
Entry for principal kudu/127.2.2.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab.
I20260619 14:32:01.558005 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.2:0
--local_ip_for_outbound_sockets=127.2.2.2
--webserver_interface=127.2.2.2
--webserver_port=0
--tserver_master_addrs=127.2.2.62:34767
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab
--principal=kudu/127.2.2.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:33365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:32:01.681380 3386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:01.681738 3386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:01.681787 3386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:01.685994 3386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:01.686070 3386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:01.686112 3386 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 14:32:01.686208 3386 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.2
I20260619 14:32:01.691648 3386 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:33365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab
--principal=kudu/127.2.2.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.2.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.2.62:34767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.2
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:32:01.692709 3386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:01.693691 3386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:32:01.704178 3386 server_base.cc:1034] running on GCE node
W20260619 14:32:01.704137 3392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:01.704226 3391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:01.704137 3394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:01.704761 3386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:01.705538 3386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:01.706702 3386 hybrid_clock.cc:648] HybridClock initialized: now 1781879521706683 us; error 35 us; skew 500 ppm
Jun 19 14:32:01 dist-test-slave-70rm krb5kdc[3129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879521, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:32:01.709589 3386 init.cc:377] Logged in from keytab as kudu/127.2.2.2@KRBTEST.COM (short username kudu)
I20260619 14:32:01.710544 3386 webserver.cc:460] Webserver started at http://127.2.2.2:34455/ using document root <none> and password file <none>
I20260619 14:32:01.711115 3386 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:01.711176 3386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:01.711413 3386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:32:01.713517 3386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/ts-1/data/instance:
uuid: "6f74b031fcce43f08ba4a30b4b2c4536"
format_stamp: "Formatted at 2026-06-19 14:32:01 on dist-test-slave-70rm"
server_key: "6af773d7f05a536fbc805d72aafc1dee"
server_key_iv: "fe38f3282028f65a393b20b394250314"
server_key_version: "encryptionkey@0"
I20260619 14:32:01.714088 3386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/ts-1/wal/instance:
uuid: "6f74b031fcce43f08ba4a30b4b2c4536"
format_stamp: "Formatted at 2026-06-19 14:32:01 on dist-test-slave-70rm"
server_key: "6af773d7f05a536fbc805d72aafc1dee"
server_key_iv: "fe38f3282028f65a393b20b394250314"
server_key_version: "encryptionkey@0"
I20260619 14:32:01.717885 3386 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:32:01.720321 3401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:01.721529 3386 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:32:01.721650 3386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/ts-1/wal
uuid: "6f74b031fcce43f08ba4a30b4b2c4536"
format_stamp: "Formatted at 2026-06-19 14:32:01 on dist-test-slave-70rm"
server_key: "6af773d7f05a536fbc805d72aafc1dee"
server_key_iv: "fe38f3282028f65a393b20b394250314"
server_key_version: "encryptionkey@0"
I20260619 14:32:01.721757 3386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:01.748347 3386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:01.751803 3386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:32:01.752000 3386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:01.753558 3386 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:32:01.753631 3386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:01.753706 3386 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:32:01.753746 3386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 14:32:01 dist-test-slave-70rm krb5kdc[3129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879521, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.2@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:32:01.764534 3386 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.2:42517
I20260619 14:32:01.765136 3516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.2:42517 every 8 connection(s)
I20260619 14:32:01.765688 3386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
I20260619 14:32:01.767808 3223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:01.754473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:46641 (local address 127.2.2.62:34767)
0619 14:32:01.754630 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:01.754636 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:01.755632 (+ 996us) server_negotiation.cc:407] Connection header received
0619 14:32:01.756844 (+ 1212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:01.756850 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:01.756918 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:01.757040 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:01.758585 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:01.759143 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:01.760358 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:01.760772 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:01.760907 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:01.763719 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:01.763723 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:01.763726 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:01.763756 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:01.765620 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:01.766301 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:01.766305 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:01.766307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:01.766409 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:01.766804 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:01.766807 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:01.766808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:01.766996 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:01.767076 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:01.767583 (+ 507us) server_negotiation.cc:299] Negotiation successful
0619 14:32:01.767690 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 14:32:01.768710 3410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:01.754975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:34767 (local address 127.2.2.2:46641)
0619 14:32:01.755455 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:01.755510 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:01.756587 (+ 1077us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:01.757227 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:01.757242 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:01.757719 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:01.758414 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:01.758437 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:01.759277 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:01.759280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:01.760207 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:01.760217 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:01.760901 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:01.760963 (+ 62us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:01.761191 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:01.761840 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:01.761858 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:01.763523 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:01.765761 (+ 2238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:01.765769 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:01.765785 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:01.766176 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:01.766521 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:01.766526 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:01.766528 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:01.766705 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:01.767115 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:01.767120 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:01.767351 (+ 231us) client_negotiation.cc:769] Sending connection context
0619 14:32:01.767507 (+ 156us) client_negotiation.cc:240] Negotiation successful
0619 14:32:01.767762 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":133,"threads_started":1}
I20260619 14:32:01.770236 3517 heartbeater.cc:347] Connected to a master server at 127.2.2.62:34767
I20260619 14:32:01.770421 3517 heartbeater.cc:464] Registering TS with master...
I20260619 14:32:01.770987 3517 heartbeater.cc:511] Master 127.2.2.62:34767 requested a full tablet report, sending...
I20260619 14:32:01.772125 3172 ts_manager.cc:194] Registered new tserver with Master: 6f74b031fcce43f08ba4a30b4b2c4536 (127.2.2.2:42517)
I20260619 14:32:01.772902 3172 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.2.2@KRBTEST.COM'} at 127.2.2.2:46641
I20260619 14:32:01.775399 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 3386
I20260619 14:32:01.775527 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/ts-1/wal/instance
I20260619 14:32:01.775847 2056 external_mini_cluster.cc:1428] Setting key 40dd59fdda70794596aa775880d637c4
I20260619 14:32:01.780462 3523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:01.772272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:34767 (local address 127.2.2.2:41133)
0619 14:32:01.772587 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:01.772607 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:01.772692 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:01.772975 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:01.772983 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:01.773245 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:01.773379 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:01.773386 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:01.774227 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:01.774231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:01.775059 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:01.775069 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:01.775496 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:01.775520 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:01.775634 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:01.776180 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:01.776195 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:01.776681 (+ 486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:01.778515 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:01.778522 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:01.778528 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:01.778824 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:01.779185 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:01.779188 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:01.779189 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:01.779286 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:01.779834 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:01.779837 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:01.779930 (+ 93us) client_negotiation.cc:769] Sending connection context
0619 14:32:01.780054 (+ 124us) client_negotiation.cc:240] Negotiation successful
0619 14:32:01.780190 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":128,"threads_started":1}
I20260619 14:32:01.780527 3223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:01.772319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:41133 (local address 127.2.2.62:34767)
0619 14:32:01.772450 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:01.772454 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:01.772674 (+ 220us) server_negotiation.cc:407] Connection header received
0619 14:32:01.772773 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:01.772776 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:01.772818 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:01.772897 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:01.773587 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:01.774075 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:01.775189 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:01.775394 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:01.775467 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:01.776806 (+ 1339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:01.776809 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:01.776812 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:01.776834 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:01.778403 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:01.778943 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:01.778949 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:01.778951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:01.779055 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:01.779382 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:01.779388 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:01.779391 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:01.779709 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:01.779828 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:01.780144 (+ 316us) server_negotiation.cc:299] Negotiation successful
0619 14:32:01.780284 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
WARNING: no policy specified for kudu/127.2.2.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab.
Entry for principal kudu/127.2.2.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab.
I20260619 14:32:01.805775 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.3:0
--local_ip_for_outbound_sockets=127.2.2.3
--webserver_interface=127.2.2.3
--webserver_port=0
--tserver_master_addrs=127.2.2.62:34767
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab
--principal=kudu/127.2.2.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:33365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:32:01.920269 3526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:01.920624 3526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:01.920702 3526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:01.925195 3526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:01.925289 3526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:01.925359 3526 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 14:32:01.925441 3526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.3
I20260619 14:32:01.931124 3526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:33365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab
--principal=kudu/127.2.2.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.2.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.2.62:34767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.3
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:32:01.932273 3526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:01.933362 3526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:32:01.943434 3532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:01.943481 3526 server_base.cc:1034] running on GCE node
W20260619 14:32:01.943420 3531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:01.943424 3534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:01.944070 3526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:01.944736 3526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:01.945955 3526 hybrid_clock.cc:648] HybridClock initialized: now 1781879521945935 us; error 41 us; skew 500 ppm
Jun 19 14:32:01 dist-test-slave-70rm krb5kdc[3129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879521, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:32:01.949131 3526 init.cc:377] Logged in from keytab as kudu/127.2.2.3@KRBTEST.COM (short username kudu)
I20260619 14:32:01.950161 3526 webserver.cc:460] Webserver started at http://127.2.2.3:42137/ using document root <none> and password file <none>
I20260619 14:32:01.950762 3526 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:01.950829 3526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:01.951017 3526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:32:01.953209 3526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/ts-2/data/instance:
uuid: "76c1a6b944af4ee89210b9c82dffc931"
format_stamp: "Formatted at 2026-06-19 14:32:01 on dist-test-slave-70rm"
server_key: "fbfca25cd85a36706968f9a0b8526970"
server_key_iv: "91aa8eb2a30dc191ce93bea673ab57bd"
server_key_version: "encryptionkey@0"
I20260619 14:32:01.953768 3526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/ts-2/wal/instance:
uuid: "76c1a6b944af4ee89210b9c82dffc931"
format_stamp: "Formatted at 2026-06-19 14:32:01 on dist-test-slave-70rm"
server_key: "fbfca25cd85a36706968f9a0b8526970"
server_key_iv: "91aa8eb2a30dc191ce93bea673ab57bd"
server_key_version: "encryptionkey@0"
I20260619 14:32:01.957751 3526 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 14:32:01.960530 3541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:01.961791 3526 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:32:01.961928 3526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/ts-2/wal
uuid: "76c1a6b944af4ee89210b9c82dffc931"
format_stamp: "Formatted at 2026-06-19 14:32:01 on dist-test-slave-70rm"
server_key: "fbfca25cd85a36706968f9a0b8526970"
server_key_iv: "91aa8eb2a30dc191ce93bea673ab57bd"
server_key_version: "encryptionkey@0"
I20260619 14:32:01.962046 3526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:01.987227 3526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:01.990738 3526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:32:01.990950 3526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:01.992439 3526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:32:01.992512 3526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:01.992585 3526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:32:01.992635 3526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 14:32:02 dist-test-slave-70rm krb5kdc[3129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879521, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.3@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:32:02.003779 3526 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.3:44499
I20260619 14:32:02.003901 3656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.3:44499 every 8 connection(s)
I20260619 14:32:02.004854 3526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
I20260619 14:32:02.007341 3223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:01.993556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.3:49889 (local address 127.2.2.62:34767)
0619 14:32:01.993695 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:01.993706 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:01.994887 (+ 1181us) server_negotiation.cc:407] Connection header received
0619 14:32:01.996071 (+ 1184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:01.996074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:01.996136 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:01.996247 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:01.998048 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:01.998738 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:02.000037 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:02.000267 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:02.000381 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:02.003225 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:02.003229 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:02.003232 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:02.003267 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:02.004982 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:02.005635 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:02.005639 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:02.005640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:02.005734 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:02.006210 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:02.006214 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:02.006216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:02.006415 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:02.006524 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:02.007029 (+ 505us) server_negotiation.cc:299] Negotiation successful
0619 14:32:02.007170 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 14:32:02.008152 3550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:01.994165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:34767 (local address 127.2.2.3:49889)
0619 14:32:01.994713 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:01.994759 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:01.995843 (+ 1084us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:01.996500 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:01.996511 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:01.997052 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:01.997846 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:01.997873 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:01.998882 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:01.998885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:01.999899 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:01.999909 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:02.000374 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:02.000438 (+ 64us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:02.000620 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:02.001318 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:02.001336 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:02.003017 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:02.005122 (+ 2105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:02.005128 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:02.005144 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:02.005505 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:02.005842 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:02.005845 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:02.005876 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:02.006072 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:02.006521 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:02.006528 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:02.006790 (+ 262us) client_negotiation.cc:769] Sending connection context
0619 14:32:02.006977 (+ 187us) client_negotiation.cc:240] Negotiation successful
0619 14:32:02.007313 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":142,"threads_started":1}
I20260619 14:32:02.009372 3657 heartbeater.cc:347] Connected to a master server at 127.2.2.62:34767
I20260619 14:32:02.009513 3657 heartbeater.cc:464] Registering TS with master...
I20260619 14:32:02.010100 3657 heartbeater.cc:511] Master 127.2.2.62:34767 requested a full tablet report, sending...
I20260619 14:32:02.011250 3173 ts_manager.cc:194] Registered new tserver with Master: 76c1a6b944af4ee89210b9c82dffc931 (127.2.2.3:44499)
I20260619 14:32:02.012003 3173 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.2.3@KRBTEST.COM'} at 127.2.2.3:49889
I20260619 14:32:02.012998 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 3526
I20260619 14:32:02.013139 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/ts-2/wal/instance
I20260619 14:32:02.013411 2056 external_mini_cluster.cc:1428] Setting key d1d68876f2701c5a4342d38a9278435a
I20260619 14:32:02.015374 2056 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:32:02.019387 3223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:02.011715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.3:53077 (local address 127.2.2.62:34767)
0619 14:32:02.011841 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:02.011845 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:02.011999 (+ 154us) server_negotiation.cc:407] Connection header received
0619 14:32:02.012136 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:02.012140 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:02.012185 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:02.012283 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:02.012725 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:02.013477 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:02.014729 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:02.014931 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:02.015008 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:02.016175 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:02.016179 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:02.016182 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:02.016206 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:02.017768 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:02.018236 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:02.018241 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:02.018243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:02.018328 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:02.018596 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:02.018599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:02.018601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:02.018771 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:02.018868 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:02.019126 (+ 258us) server_negotiation.cc:299] Negotiation successful
0619 14:32:02.019228 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 14:32:02.019404 3663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:02.011654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:34767 (local address 127.2.2.3:53077)
0619 14:32:02.011916 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:02.011932 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:02.012034 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:02.012308 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:02.012311 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:02.012485 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:02.012615 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:02.012623 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:02.013603 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:02.013606 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:02.014632 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:02.014642 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:02.015016 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:02.015033 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:02.015137 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:02.015699 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:02.015717 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:02.016029 (+ 312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:02.017895 (+ 1866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:02.017898 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:02.017900 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:02.018103 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:02.018420 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:02.018422 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:02.018424 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:02.018507 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:02.018868 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:02.018872 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:02.018964 (+ 92us) client_negotiation.cc:769] Sending connection context
0619 14:32:02.019061 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 14:32:02.019227 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":156,"thread_start_us":100,"threads_started":1}
Jun 19 14:32:02 dist-test-slave-70rm krb5kdc[3129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879522, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 14:32:02 dist-test-slave-70rm krb5kdc[3129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879522, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:32:02.043340 3223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:02.034941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33038 (local address 127.2.2.62:34767)
0619 14:32:02.035078 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:02.035081 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:02.035097 (+ 16us) server_negotiation.cc:407] Connection header received
0619 14:32:02.035143 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:02.035146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:02.035188 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:02.035273 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:02.035983 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:02.036520 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:02.037446 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:02.037672 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:02.037766 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:02.040403 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:02.040406 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:02.040409 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:02.040436 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:02.041889 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:02.042381 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:02.042384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:02.042385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:02.042459 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:02.042682 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:02.042684 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:02.042685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:02.042842 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:02.042933 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:02.043123 (+ 190us) server_negotiation.cc:299] Negotiation successful
0619 14:32:02.043218 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 14:32:02.046406 3173 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33038:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:32:02.049413 3173 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:32:02.064450 3674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:02.058713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.2:42517 (local address 127.0.0.1:48182)
0619 14:32:02.059058 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:02.059071 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:02.059149 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:02.060572 (+ 1423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:02.060575 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:02.060591 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:32:02.060680 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:02.060686 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:02.061890 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:02.061895 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:02.062953 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:02.062962 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:02.063871 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:02.063874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:02.063986 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:02.064023 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 14:32:02.064168 (+ 145us) client_negotiation.cc:240] Negotiation successful
0619 14:32:02.064299 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":5,"thread_start_us":101,"threads_started":1}
I20260619 14:32:02.064459 3679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:02.059251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48182 (local address 127.2.2.2:42517)
0619 14:32:02.059833 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:02.059846 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:02.059870 (+ 24us) server_negotiation.cc:407] Connection header received
0619 14:32:02.059976 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:02.059983 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:02.060273 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:02.060422 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:32:02.060806 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:02.061781 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:02.063109 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:02.063747 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:02.063870 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:02.064050 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:02.064198 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 14:32:02.064296 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":96,"threads_started":1}
I20260619 14:32:02.064693 3677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:02.058204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.3:44499 (local address 127.0.0.1:36328)
0619 14:32:02.058939 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:02.058988 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:02.059088 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:02.059925 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:02.059928 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:02.059946 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:32:02.060125 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:02.060132 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:02.061417 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:02.061423 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:02.063172 (+ 1749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:02.063183 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:02.064275 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:02.064279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:02.064376 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:02.064398 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 14:32:02.064462 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 14:32:02.064534 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"mutex_wait_us":209,"thread_start_us":126,"threads_started":1}
I20260619 14:32:02.064889 3676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:02.058010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.1:35249 (local address 127.0.0.1:33980)
0619 14:32:02.058711 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:02.058757 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:02.058866 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:02.059855 (+ 989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:02.059861 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:02.059882 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:32:02.060090 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:02.060099 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:02.061539 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:02.061542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:02.063172 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:02.063183 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:02.064453 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:02.064456 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:02.064533 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:02.064551 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 14:32:02.064614 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 14:32:02.064670 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"mutex_wait_us":130,"thread_start_us":108,"threads_started":1}
I20260619 14:32:02.065030 3675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:02.058217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33980 (local address 127.2.2.1:35249)
0619 14:32:02.059378 (+ 1161us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:02.059385 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:02.059409 (+ 24us) server_negotiation.cc:407] Connection header received
0619 14:32:02.059485 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:02.059489 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:02.059690 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:02.060254 (+ 564us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:32:02.060424 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:02.061432 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:02.063320 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:02.064135 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:02.064254 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:02.064395 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:02.064766 (+ 371us) server_negotiation.cc:299] Negotiation successful
0619 14:32:02.064872 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1061,"thread_start_us":116,"threads_started":1}
I20260619 14:32:02.065430 3678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:02.059132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36328 (local address 127.2.2.3:44499)
0619 14:32:02.059458 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:02.059470 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:02.059492 (+ 22us) server_negotiation.cc:407] Connection header received
0619 14:32:02.059565 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:02.059570 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:02.059766 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:02.060289 (+ 523us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:32:02.060439 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:02.061308 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:02.063320 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:02.064142 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:02.064997 (+ 855us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:02.065168 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:02.065215 (+ 47us) server_negotiation.cc:299] Negotiation successful
0619 14:32:02.065305 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":119,"threads_started":1}
I20260619 14:32:02.067945 3311 tablet_service.cc:1467] Processing CreateTablet for tablet 464be2061a6947a18a00cdc9e31c6a0d (DEFAULT_TABLE table=test-table [id=3deb19cb6fd14df5ab05f74ac8ca0ad8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:32:02.067956 3591 tablet_service.cc:1467] Processing CreateTablet for tablet 464be2061a6947a18a00cdc9e31c6a0d (DEFAULT_TABLE table=test-table [id=3deb19cb6fd14df5ab05f74ac8ca0ad8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:32:02.068846 3311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 464be2061a6947a18a00cdc9e31c6a0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:02.068852 3591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 464be2061a6947a18a00cdc9e31c6a0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:02.072017 3451 tablet_service.cc:1467] Processing CreateTablet for tablet 464be2061a6947a18a00cdc9e31c6a0d (DEFAULT_TABLE table=test-table [id=3deb19cb6fd14df5ab05f74ac8ca0ad8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:32:02.073292 3451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 464be2061a6947a18a00cdc9e31c6a0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:02.076129 3680 tablet_bootstrap.cc:492] T 464be2061a6947a18a00cdc9e31c6a0d P fa0316c331a843ba89f6c3258d151bca: Bootstrap starting.
I20260619 14:32:02.077566 3681 tablet_bootstrap.cc:492] T 464be2061a6947a18a00cdc9e31c6a0d P 76c1a6b944af4ee89210b9c82dffc931: Bootstrap starting.
I20260619 14:32:02.078585 3680 tablet_bootstrap.cc:654] T 464be2061a6947a18a00cdc9e31c6a0d P fa0316c331a843ba89f6c3258d151bca: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:02.079497 3680 log.cc:824] T 464be2061a6947a18a00cdc9e31c6a0d P fa0316c331a843ba89f6c3258d151bca: Log is configured to *not* fsync() on all Append() calls
I20260619 14:32:02.079761 3681 tablet_bootstrap.cc:654] T 464be2061a6947a18a00cdc9e31c6a0d P 76c1a6b944af4ee89210b9c82dffc931: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:02.080528 3681 log.cc:824] T 464be2061a6947a18a00cdc9e31c6a0d P 76c1a6b944af4ee89210b9c82dffc931: Log is configured to *not* fsync() on all Append() calls
I20260619 14:32:02.081491 3683 tablet_bootstrap.cc:492] T 464be2061a6947a18a00cdc9e31c6a0d P 6f74b031fcce43f08ba4a30b4b2c4536: Bootstrap starting.
I20260619 14:32:02.082486 3681 tablet_bootstrap.cc:492] T 464be2061a6947a18a00cdc9e31c6a0d P 76c1a6b944af4ee89210b9c82dffc931: No bootstrap required, opened a new log
I20260619 14:32:02.082486 3680 tablet_bootstrap.cc:492] T 464be2061a6947a18a00cdc9e31c6a0d P fa0316c331a843ba89f6c3258d151bca: No bootstrap required, opened a new log
I20260619 14:32:02.082692 3680 ts_tablet_manager.cc:1397] T 464be2061a6947a18a00cdc9e31c6a0d P fa0316c331a843ba89f6c3258d151bca: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 14:32:02.082692 3681 ts_tablet_manager.cc:1397] T 464be2061a6947a18a00cdc9e31c6a0d P 76c1a6b944af4ee89210b9c82dffc931: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:32:02.083523 3683 tablet_bootstrap.cc:654] T 464be2061a6947a18a00cdc9e31c6a0d P 6f74b031fcce43f08ba4a30b4b2c4536: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:02.084218 3683 log.cc:824] T 464be2061a6947a18a00cdc9e31c6a0d P 6f74b031fcce43f08ba4a30b4b2c4536: Log is configured to *not* fsync() on all Append() calls
I20260619 14:32:02.085942 3683 tablet_bootstrap.cc:492] T 464be2061a6947a18a00cdc9e31c6a0d P 6f74b031fcce43f08ba4a30b4b2c4536: No bootstrap required, opened a new log
I20260619 14:32:02.086189 3683 ts_tablet_manager.cc:1397] T 464be2061a6947a18a00cdc9e31c6a0d P 6f74b031fcce43f08ba4a30b4b2c4536: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260619 14:32:02.086848 3681 raft_consensus.cc:348] T 464be2061a6947a18a00cdc9e31c6a0d P 76c1a6b944af4ee89210b9c82dffc931 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa0316c331a843ba89f6c3258d151bca" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 35249 } } peers { permanent_uuid: "76c1a6b944af4ee89210b9c82dffc931" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44499 } } peers { permanent_uuid: "6f74b031fcce43f08ba4a30b4b2c4536" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 42517 } }
I20260619 14:32:02.087122 3681 raft_consensus.cc:374] T 464be2061a6947a18a00cdc9e31c6a0d P 76c1a6b944af4ee89210b9c82dffc931 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:02.087188 3681 raft_consensus.cc:729] T 464be2061a6947a18a00cdc9e31c6a0d P 76c1a6b944af4ee89210b9c82dffc931 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c1a6b944af4ee89210b9c82dffc931, State: Initialized, Role: FOLLOWER
I20260619 14:32:02.087225 3680 raft_consensus.cc:348] T 464be2061a6947a18a00cdc9e31c6a0d P fa0316c331a843ba89f6c3258d151bca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa0316c331a843ba89f6c3258d151bca" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 35249 } } peers { permanent_uuid: "76c1a6b944af4ee89210b9c82dffc931" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44499 } } peers { permanent_uuid: "6f74b031fcce43f08ba4a30b4b2c4536" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 42517 } }
I20260619 14:32:02.087682 3680 raft_consensus.cc:374] T 464be2061a6947a18a00cdc9e31c6a0d P fa0316c331a843ba89f6c3258d151bca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:02.087731 3680 raft_consensus.cc:729] T 464be2061a6947a18a00cdc9e31c6a0d P fa0316c331a843ba89f6c3258d151bca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa0316c331a843ba89f6c3258d151bca, State: Initialized, Role: FOLLOWER
I20260619 14:32:02.087641 3681 consensus_queue.cc:260] T 464be2061a6947a18a00cdc9e31c6a0d P 76c1a6b944af4ee89210b9c82dffc931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa0316c331a843ba89f6c3258d151bca" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 35249 } } peers { permanent_uuid: "76c1a6b944af4ee89210b9c82dffc931" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44499 } } peers { permanent_uuid: "6f74b031fcce43f08ba4a30b4b2c4536" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 42517 } }
I20260619 14:32:02.088537 3680 consensus_queue.cc:260] T 464be2061a6947a18a00cdc9e31c6a0d P fa0316c331a843ba89f6c3258d151bca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa0316c331a843ba89f6c3258d151bca" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 35249 } } peers { permanent_uuid: "76c1a6b944af4ee89210b9c82dffc931" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44499 } } peers { permanent_uuid: "6f74b031fcce43f08ba4a30b4b2c4536" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 42517 } }
I20260619 14:32:02.088673 3681 ts_tablet_manager.cc:1428] T 464be2061a6947a18a00cdc9e31c6a0d P 76c1a6b944af4ee89210b9c82dffc931: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 14:32:02.089020 3657 heartbeater.cc:503] Master 127.2.2.62:34767 was elected leader, sending a full tablet report...
I20260619 14:32:02.089351 3680 ts_tablet_manager.cc:1428] T 464be2061a6947a18a00cdc9e31c6a0d P fa0316c331a843ba89f6c3258d151bca: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260619 14:32:02.089653 3377 heartbeater.cc:503] Master 127.2.2.62:34767 was elected leader, sending a full tablet report...
I20260619 14:32:02.089865 3683 raft_consensus.cc:348] T 464be2061a6947a18a00cdc9e31c6a0d P 6f74b031fcce43f08ba4a30b4b2c4536 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa0316c331a843ba89f6c3258d151bca" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 35249 } } peers { permanent_uuid: "76c1a6b944af4ee89210b9c82dffc931" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44499 } } peers { permanent_uuid: "6f74b031fcce43f08ba4a30b4b2c4536" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 42517 } }
I20260619 14:32:02.090152 3683 raft_consensus.cc:374] T 464be2061a6947a18a00cdc9e31c6a0d P 6f74b031fcce43f08ba4a30b4b2c4536 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:02.090219 3683 raft_consensus.cc:729] T 464be2061a6947a18a00cdc9e31c6a0d P 6f74b031fcce43f08ba4a30b4b2c4536 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f74b031fcce43f08ba4a30b4b2c4536, State: Initialized, Role: FOLLOWER
I20260619 14:32:02.090708 3683 consensus_queue.cc:260] T 464be2061a6947a18a00cdc9e31c6a0d P 6f74b031fcce43f08ba4a30b4b2c4536 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa0316c331a843ba89f6c3258d151bca" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 35249 } } peers { permanent_uuid: "76c1a6b944af4ee89210b9c82dffc931" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44499 } } peers { permanent_uuid: "6f74b031fcce43f08ba4a30b4b2c4536" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 42517 } }
I20260619 14:32:02.091358 3517 heartbeater.cc:503] Master 127.2.2.62:34767 was elected leader, sending a full tablet report...
I20260619 14:32:02.091704 3683 ts_tablet_manager.cc:1428] T 464be2061a6947a18a00cdc9e31c6a0d P 6f74b031fcce43f08ba4a30b4b2c4536: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 14:32:02.186255 3686 raft_consensus.cc:482] T 464be2061a6947a18a00cdc9e31c6a0d P 76c1a6b944af4ee89210b9c82dffc931 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:32:02.186467 3686 raft_consensus.cc:504] T 464be2061a6947a18a00cdc9e31c6a0d P 76c1a6b944af4ee89210b9c82dffc931 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa0316c331a843ba89f6c3258d151bca" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 35249 } } peers { permanent_uuid: "76c1a6b944af4ee89210b9c82dffc931" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44499 } } peers { permanent_uuid: "6f74b031fcce43f08ba4a30b4b2c4536" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 42517 } }
I20260619 14:32:02.187630 3686 leader_election.cc:288] T 464be2061a6947a18a00cdc9e31c6a0d P 76c1a6b944af4ee89210b9c82dffc931 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa0316c331a843ba89f6c3258d151bca (127.2.2.1:35249), 6f74b031fcce43f08ba4a30b4b2c4536 (127.2.2.2:42517)
I20260619 14:32:02.191399 3675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:02.188051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.3:36373 (local address 127.2.2.1:35249)
0619 14:32:02.188201 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:02.188207 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:02.188226 (+ 19us) server_negotiation.cc:407] Connection header received
0619 14:32:02.188287 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:02.188293 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:02.188350 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:02.188449 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:32:02.188728 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:02.189243 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:02.190345 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:02.190797 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:02.190923 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:02.190993 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:02.191188 (+ 195us) server_negotiation.cc:299] Negotiation successful
0619 14:32:02.191263 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 14:32:02.191399 3550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:02.187881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.1:35249 (local address 127.2.2.3:36373)
0619 14:32:02.188011 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:02.188028 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:02.188159 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:02.188463 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:02.188465 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:02.188483 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:32:02.188610 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:02.188616 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:02.189357 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:02.189362 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:02.190211 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:02.190220 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:02.190930 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:02.190934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:02.191064 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:02.191084 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 14:32:02.191172 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 14:32:02.191256 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260619 14:32:02.192070 3689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:02.188017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.2:42517 (local address 127.2.2.3:39787)
0619 14:32:02.188412 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:02.188423 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:02.188544 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:02.188798 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:02.188801 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:02.188820 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:32:02.188945 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:02.188950 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:02.189666 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:02.189672 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:02.190930 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:02.190939 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:02.191690 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:02.191696 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:02.191793 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:02.191817 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 14:32:02.191896 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 14:32:02.191958 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":147,"threads_started":1}
I20260619 14:32:02.192203 3679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:02.188181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.3:39787 (local address 127.2.2.2:42517)
0619 14:32:02.188309 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:02.188315 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:02.188503 (+ 188us) server_negotiation.cc:407] Connection header received
0619 14:32:02.188655 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:02.188663 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:02.188706 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:02.188788 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:32:02.189050 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:02.189546 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:02.191060 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:02.191456 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:02.191609 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:02.191678 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:02.191976 (+ 298us) server_negotiation.cc:299] Negotiation successful
0619 14:32:02.192081 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 14:32:02.192265 3331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "464be2061a6947a18a00cdc9e31c6a0d" candidate_uuid: "76c1a6b944af4ee89210b9c82dffc931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa0316c331a843ba89f6c3258d151bca" is_pre_election: true
I20260619 14:32:02.192646 3331 raft_consensus.cc:2454] T 464be2061a6947a18a00cdc9e31c6a0d P fa0316c331a843ba89f6c3258d151bca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c1a6b944af4ee89210b9c82dffc931 in term 0.
I20260619 14:32:02.192816 3471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "464be2061a6947a18a00cdc9e31c6a0d" candidate_uuid: "76c1a6b944af4ee89210b9c82dffc931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f74b031fcce43f08ba4a30b4b2c4536" is_pre_election: true
I20260619 14:32:02.193135 3471 raft_consensus.cc:2454] T 464be2061a6947a18a00cdc9e31c6a0d P 6f74b031fcce43f08ba4a30b4b2c4536 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c1a6b944af4ee89210b9c82dffc931 in term 0.
I20260619 14:32:02.193212 3544 leader_election.cc:302] T 464be2061a6947a18a00cdc9e31c6a0d P 76c1a6b944af4ee89210b9c82dffc931 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76c1a6b944af4ee89210b9c82dffc931, fa0316c331a843ba89f6c3258d151bca; no voters:
I20260619 14:32:02.193507 3686 raft_consensus.cc:2781] T 464be2061a6947a18a00cdc9e31c6a0d P 76c1a6b944af4ee89210b9c82dffc931 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:32:02.193602 3686 raft_consensus.cc:482] T 464be2061a6947a18a00cdc9e31c6a0d P 76c1a6b944af4ee89210b9c82dffc931 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:32:02.193634 3686 raft_consensus.cc:3037] T 464be2061a6947a18a00cdc9e31c6a0d P 76c1a6b944af4ee89210b9c82dffc931 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:02.194763 3686 raft_consensus.cc:504] T 464be2061a6947a18a00cdc9e31c6a0d P 76c1a6b944af4ee89210b9c82dffc931 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa0316c331a843ba89f6c3258d151bca" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 35249 } } peers { permanent_uuid: "76c1a6b944af4ee89210b9c82dffc931" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44499 } } peers { permanent_uuid: "6f74b031fcce43f08ba4a30b4b2c4536" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 42517 } }
I20260619 14:32:02.195166 3686 leader_election.cc:288] T 464be2061a6947a18a00cdc9e31c6a0d P 76c1a6b944af4ee89210b9c82dffc931 [CANDIDATE]: Term 1 election: Requested vote from peers fa0316c331a843ba89f6c3258d151bca (127.2.2.1:35249), 6f74b031fcce43f08ba4a30b4b2c4536 (127.2.2.2:42517)
I20260619 14:32:02.195583 3331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "464be2061a6947a18a00cdc9e31c6a0d" candidate_uuid: "76c1a6b944af4ee89210b9c82dffc931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa0316c331a843ba89f6c3258d151bca"
I20260619 14:32:02.195698 3331 raft_consensus.cc:3037] T 464be2061a6947a18a00cdc9e31c6a0d P fa0316c331a843ba89f6c3258d151bca [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:02.196678 3331 raft_consensus.cc:2454] T 464be2061a6947a18a00cdc9e31c6a0d P fa0316c331a843ba89f6c3258d151bca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c1a6b944af4ee89210b9c82dffc931 in term 1.
I20260619 14:32:02.196950 3471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "464be2061a6947a18a00cdc9e31c6a0d" candidate_uuid: "76c1a6b944af4ee89210b9c82dffc931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f74b031fcce43f08ba4a30b4b2c4536"
I20260619 14:32:02.196975 3544 leader_election.cc:302] T 464be2061a6947a18a00cdc9e31c6a0d P 76c1a6b944af4ee89210b9c82dffc931 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76c1a6b944af4ee89210b9c82dffc931, fa0316c331a843ba89f6c3258d151bca; no voters:
I20260619 14:32:02.197088 3471 raft_consensus.cc:3037] T 464be2061a6947a18a00cdc9e31c6a0d P 6f74b031fcce43f08ba4a30b4b2c4536 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:02.197171 3686 raft_consensus.cc:2781] T 464be2061a6947a18a00cdc9e31c6a0d P 76c1a6b944af4ee89210b9c82dffc931 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:32:02.197413 3686 raft_consensus.cc:686] T 464be2061a6947a18a00cdc9e31c6a0d P 76c1a6b944af4ee89210b9c82dffc931 [term 1 LEADER]: Becoming Leader. State: Replica: 76c1a6b944af4ee89210b9c82dffc931, State: Running, Role: LEADER
I20260619 14:32:02.197793 3686 consensus_queue.cc:237] T 464be2061a6947a18a00cdc9e31c6a0d P 76c1a6b944af4ee89210b9c82dffc931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa0316c331a843ba89f6c3258d151bca" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 35249 } } peers { permanent_uuid: "76c1a6b944af4ee89210b9c82dffc931" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44499 } } peers { permanent_uuid: "6f74b031fcce43f08ba4a30b4b2c4536" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 42517 } }
I20260619 14:32:02.198909 3471 raft_consensus.cc:2454] T 464be2061a6947a18a00cdc9e31c6a0d P 6f74b031fcce43f08ba4a30b4b2c4536 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c1a6b944af4ee89210b9c82dffc931 in term 1.
I20260619 14:32:02.201463 3172 catalog_manager.cc:5495] T 464be2061a6947a18a00cdc9e31c6a0d P 76c1a6b944af4ee89210b9c82dffc931 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76c1a6b944af4ee89210b9c82dffc931 (127.2.2.3). New cstate: current_term: 1 leader_uuid: "76c1a6b944af4ee89210b9c82dffc931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa0316c331a843ba89f6c3258d151bca" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 35249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76c1a6b944af4ee89210b9c82dffc931" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f74b031fcce43f08ba4a30b4b2c4536" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 42517 } health_report { overall_health: UNKNOWN } } }
I20260619 14:32:02.223927 3678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:02.219815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36334 (local address 127.2.2.3:44499)
0619 14:32:02.219976 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:02.219979 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:02.219994 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:32:02.220069 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:02.220071 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:02.220148 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:02.220243 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:32:02.220522 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:02.221005 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:02.222300 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:02.222626 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:02.222713 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:02.223074 (+ 361us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:32:02.223511 (+ 437us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:32:02.223598 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:02.223705 (+ 107us) server_negotiation.cc:299] Negotiation successful
0619 14:32:02.223759 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 14:32:02.230489 3331 raft_consensus.cc:1261] T 464be2061a6947a18a00cdc9e31c6a0d P fa0316c331a843ba89f6c3258d151bca [term 1 FOLLOWER]: Refusing update from remote peer 76c1a6b944af4ee89210b9c82dffc931: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:32:02.230485 3471 raft_consensus.cc:1261] T 464be2061a6947a18a00cdc9e31c6a0d P 6f74b031fcce43f08ba4a30b4b2c4536 [term 1 FOLLOWER]: Refusing update from remote peer 76c1a6b944af4ee89210b9c82dffc931: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:32:02.231299 3690 consensus_queue.cc:1035] T 464be2061a6947a18a00cdc9e31c6a0d P 76c1a6b944af4ee89210b9c82dffc931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f74b031fcce43f08ba4a30b4b2c4536" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 42517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:32:02.231561 3686 consensus_queue.cc:1035] T 464be2061a6947a18a00cdc9e31c6a0d P 76c1a6b944af4ee89210b9c82dffc931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa0316c331a843ba89f6c3258d151bca" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 35249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:32:02.240634 3694 mvcc.cc:205] Tried to move back new op lower bound from 7298578523048280064 to 7298578522925813760. Current Snapshot: MvccSnapshot[applied={T|T < 7298578523048280064}]
I20260619 14:32:02.241565 3695 mvcc.cc:205] Tried to move back new op lower bound from 7298578523048280064 to 7298578522925813760. Current Snapshot: MvccSnapshot[applied={T|T < 7298578523048280064}]
I20260619 14:32:02.253911 3170 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:32:02.254185 3170 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:32:02.256897 3170 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 25aefb30399c43b7a1516ee1783654fb: Sending DeleteTablet for 3 replicas of tablet 464be2061a6947a18a00cdc9e31c6a0d
I20260619 14:32:02.257858 3591 tablet_service.cc:1514] Processing DeleteTablet for tablet 464be2061a6947a18a00cdc9e31c6a0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:32:02 UTC) from {username='kudu', principal='kudu/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:36328
I20260619 14:32:02.257881 3451 tablet_service.cc:1514] Processing DeleteTablet for tablet 464be2061a6947a18a00cdc9e31c6a0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:32:02 UTC) from {username='kudu', principal='kudu/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:48182
I20260619 14:32:02.258157 3311 tablet_service.cc:1514] Processing DeleteTablet for tablet 464be2061a6947a18a00cdc9e31c6a0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:32:02 UTC) from {username='kudu', principal='kudu/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:33980
I20260619 14:32:02.258642 3705 tablet_replica.cc:330] T 464be2061a6947a18a00cdc9e31c6a0d P fa0316c331a843ba89f6c3258d151bca: stopping tablet replica
I20260619 14:32:02.258937 3703 tablet_replica.cc:330] T 464be2061a6947a18a00cdc9e31c6a0d P 76c1a6b944af4ee89210b9c82dffc931: stopping tablet replica
I20260619 14:32:02.259004 3705 raft_consensus.cc:2229] T 464be2061a6947a18a00cdc9e31c6a0d P fa0316c331a843ba89f6c3258d151bca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:32:02.258941 3170 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33038:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 14:32:02.259227 3703 raft_consensus.cc:2229] T 464be2061a6947a18a00cdc9e31c6a0d P 76c1a6b944af4ee89210b9c82dffc931 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:32:02.259339 3705 raft_consensus.cc:2258] T 464be2061a6947a18a00cdc9e31c6a0d P fa0316c331a843ba89f6c3258d151bca [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 14:32:02.259380 3170 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:32:02.259639 3703 raft_consensus.cc:2258] T 464be2061a6947a18a00cdc9e31c6a0d P 76c1a6b944af4ee89210b9c82dffc931 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:32:02.260898 3703 ts_tablet_manager.cc:1905] T 464be2061a6947a18a00cdc9e31c6a0d P 76c1a6b944af4ee89210b9c82dffc931: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:32:02.260898 3705 ts_tablet_manager.cc:1905] T 464be2061a6947a18a00cdc9e31c6a0d P fa0316c331a843ba89f6c3258d151bca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:32:02.263669 3703 ts_tablet_manager.cc:1918] T 464be2061a6947a18a00cdc9e31c6a0d P 76c1a6b944af4ee89210b9c82dffc931: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:32:02.263660 3705 ts_tablet_manager.cc:1918] T 464be2061a6947a18a00cdc9e31c6a0d P fa0316c331a843ba89f6c3258d151bca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:32:02.263769 3705 log.cc:1196] T 464be2061a6947a18a00cdc9e31c6a0d P fa0316c331a843ba89f6c3258d151bca: Deleting WAL directory at /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/ts-0/wal/wals/464be2061a6947a18a00cdc9e31c6a0d
I20260619 14:32:02.263769 3703 log.cc:1196] T 464be2061a6947a18a00cdc9e31c6a0d P 76c1a6b944af4ee89210b9c82dffc931: Deleting WAL directory at /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/ts-2/wal/wals/464be2061a6947a18a00cdc9e31c6a0d
I20260619 14:32:02.264091 3705 ts_tablet_manager.cc:1939] T 464be2061a6947a18a00cdc9e31c6a0d P fa0316c331a843ba89f6c3258d151bca: Deleting consensus metadata
I20260619 14:32:02.264091 3703 ts_tablet_manager.cc:1939] T 464be2061a6947a18a00cdc9e31c6a0d P 76c1a6b944af4ee89210b9c82dffc931: Deleting consensus metadata
I20260619 14:32:02.265362 3704 tablet_replica.cc:330] T 464be2061a6947a18a00cdc9e31c6a0d P 6f74b031fcce43f08ba4a30b4b2c4536: stopping tablet replica
I20260619 14:32:02.265576 3451 tablet_service.cc:1467] Processing CreateTablet for tablet 9dd56bf4a22d4a7dbccb152b1fde4446 (DEFAULT_TABLE table=test-table [id=db3196998e5e43178ace6c34ad883450]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:32:02.265576 3591 tablet_service.cc:1467] Processing CreateTablet for tablet 9dd56bf4a22d4a7dbccb152b1fde4446 (DEFAULT_TABLE table=test-table [id=db3196998e5e43178ace6c34ad883450]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:32:02.265722 3704 raft_consensus.cc:2229] T 464be2061a6947a18a00cdc9e31c6a0d P 6f74b031fcce43f08ba4a30b4b2c4536 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:32:02.265825 3591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dd56bf4a22d4a7dbccb152b1fde4446. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:02.265825 3451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dd56bf4a22d4a7dbccb152b1fde4446. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:02.266012 3704 raft_consensus.cc:2258] T 464be2061a6947a18a00cdc9e31c6a0d P 6f74b031fcce43f08ba4a30b4b2c4536 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:32:02.266443 3159 catalog_manager.cc:4841] TS 76c1a6b944af4ee89210b9c82dffc931 (127.2.2.3:44499): tablet 464be2061a6947a18a00cdc9e31c6a0d (table test-table [id=3deb19cb6fd14df5ab05f74ac8ca0ad8]) successfully deleted
I20260619 14:32:02.266749 3160 catalog_manager.cc:4841] TS fa0316c331a843ba89f6c3258d151bca (127.2.2.1:35249): tablet 464be2061a6947a18a00cdc9e31c6a0d (table test-table [id=3deb19cb6fd14df5ab05f74ac8ca0ad8]) successfully deleted
I20260619 14:32:02.267073 3311 tablet_service.cc:1467] Processing CreateTablet for tablet 9dd56bf4a22d4a7dbccb152b1fde4446 (DEFAULT_TABLE table=test-table [id=db3196998e5e43178ace6c34ad883450]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:32:02.267323 3311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dd56bf4a22d4a7dbccb152b1fde4446. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:02.268146 3681 tablet_bootstrap.cc:492] T 9dd56bf4a22d4a7dbccb152b1fde4446 P 76c1a6b944af4ee89210b9c82dffc931: Bootstrap starting.
I20260619 14:32:02.269207 3681 tablet_bootstrap.cc:654] T 9dd56bf4a22d4a7dbccb152b1fde4446 P 76c1a6b944af4ee89210b9c82dffc931: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:02.269755 3680 tablet_bootstrap.cc:492] T 9dd56bf4a22d4a7dbccb152b1fde4446 P fa0316c331a843ba89f6c3258d151bca: Bootstrap starting.
I20260619 14:32:02.269836 3683 tablet_bootstrap.cc:492] T 9dd56bf4a22d4a7dbccb152b1fde4446 P 6f74b031fcce43f08ba4a30b4b2c4536: Bootstrap starting.
I20260619 14:32:02.271224 3680 tablet_bootstrap.cc:654] T 9dd56bf4a22d4a7dbccb152b1fde4446 P fa0316c331a843ba89f6c3258d151bca: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:02.271224 3683 tablet_bootstrap.cc:654] T 9dd56bf4a22d4a7dbccb152b1fde4446 P 6f74b031fcce43f08ba4a30b4b2c4536: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:02.272570 3680 tablet_bootstrap.cc:492] T 9dd56bf4a22d4a7dbccb152b1fde4446 P fa0316c331a843ba89f6c3258d151bca: No bootstrap required, opened a new log
I20260619 14:32:02.272665 3680 ts_tablet_manager.cc:1397] T 9dd56bf4a22d4a7dbccb152b1fde4446 P fa0316c331a843ba89f6c3258d151bca: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260619 14:32:02.272789 3683 tablet_bootstrap.cc:492] T 9dd56bf4a22d4a7dbccb152b1fde4446 P 6f74b031fcce43f08ba4a30b4b2c4536: No bootstrap required, opened a new log
I20260619 14:32:02.272866 3683 ts_tablet_manager.cc:1397] T 9dd56bf4a22d4a7dbccb152b1fde4446 P 6f74b031fcce43f08ba4a30b4b2c4536: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 14:32:02.273289 3683 raft_consensus.cc:348] T 9dd56bf4a22d4a7dbccb152b1fde4446 P 6f74b031fcce43f08ba4a30b4b2c4536 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa0316c331a843ba89f6c3258d151bca" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 35249 } } peers { permanent_uuid: "6f74b031fcce43f08ba4a30b4b2c4536" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 42517 } } peers { permanent_uuid: "76c1a6b944af4ee89210b9c82dffc931" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44499 } }
I20260619 14:32:02.273396 3683 raft_consensus.cc:374] T 9dd56bf4a22d4a7dbccb152b1fde4446 P 6f74b031fcce43f08ba4a30b4b2c4536 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:02.273427 3683 raft_consensus.cc:729] T 9dd56bf4a22d4a7dbccb152b1fde4446 P 6f74b031fcce43f08ba4a30b4b2c4536 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f74b031fcce43f08ba4a30b4b2c4536, State: Initialized, Role: FOLLOWER
I20260619 14:32:02.273452 3680 raft_consensus.cc:348] T 9dd56bf4a22d4a7dbccb152b1fde4446 P fa0316c331a843ba89f6c3258d151bca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa0316c331a843ba89f6c3258d151bca" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 35249 } } peers { permanent_uuid: "6f74b031fcce43f08ba4a30b4b2c4536" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 42517 } } peers { permanent_uuid: "76c1a6b944af4ee89210b9c82dffc931" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44499 } }
I20260619 14:32:02.273572 3680 raft_consensus.cc:374] T 9dd56bf4a22d4a7dbccb152b1fde4446 P fa0316c331a843ba89f6c3258d151bca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:02.273608 3680 raft_consensus.cc:729] T 9dd56bf4a22d4a7dbccb152b1fde4446 P fa0316c331a843ba89f6c3258d151bca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa0316c331a843ba89f6c3258d151bca, State: Initialized, Role: FOLLOWER
I20260619 14:32:02.273634 3683 consensus_queue.cc:260] T 9dd56bf4a22d4a7dbccb152b1fde4446 P 6f74b031fcce43f08ba4a30b4b2c4536 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa0316c331a843ba89f6c3258d151bca" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 35249 } } peers { permanent_uuid: "6f74b031fcce43f08ba4a30b4b2c4536" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 42517 } } peers { permanent_uuid: "76c1a6b944af4ee89210b9c82dffc931" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44499 } }
I20260619 14:32:02.273953 3680 consensus_queue.cc:260] T 9dd56bf4a22d4a7dbccb152b1fde4446 P fa0316c331a843ba89f6c3258d151bca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa0316c331a843ba89f6c3258d151bca" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 35249 } } peers { permanent_uuid: "6f74b031fcce43f08ba4a30b4b2c4536" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 42517 } } peers { permanent_uuid: "76c1a6b944af4ee89210b9c82dffc931" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44499 } }
I20260619 14:32:02.274374 3681 tablet_bootstrap.cc:492] T 9dd56bf4a22d4a7dbccb152b1fde4446 P 76c1a6b944af4ee89210b9c82dffc931: No bootstrap required, opened a new log
I20260619 14:32:02.274472 3681 ts_tablet_manager.cc:1397] T 9dd56bf4a22d4a7dbccb152b1fde4446 P 76c1a6b944af4ee89210b9c82dffc931: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260619 14:32:02.274928 3681 raft_consensus.cc:348] T 9dd56bf4a22d4a7dbccb152b1fde4446 P 76c1a6b944af4ee89210b9c82dffc931 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa0316c331a843ba89f6c3258d151bca" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 35249 } } peers { permanent_uuid: "6f74b031fcce43f08ba4a30b4b2c4536" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 42517 } } peers { permanent_uuid: "76c1a6b944af4ee89210b9c82dffc931" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44499 } }
I20260619 14:32:02.275063 3681 raft_consensus.cc:374] T 9dd56bf4a22d4a7dbccb152b1fde4446 P 76c1a6b944af4ee89210b9c82dffc931 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:02.275104 3681 raft_consensus.cc:729] T 9dd56bf4a22d4a7dbccb152b1fde4446 P 76c1a6b944af4ee89210b9c82dffc931 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c1a6b944af4ee89210b9c82dffc931, State: Initialized, Role: FOLLOWER
I20260619 14:32:02.275013 3680 ts_tablet_manager.cc:1428] T 9dd56bf4a22d4a7dbccb152b1fde4446 P fa0316c331a843ba89f6c3258d151bca: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260619 14:32:02.275216 3683 ts_tablet_manager.cc:1428] T 9dd56bf4a22d4a7dbccb152b1fde4446 P 6f74b031fcce43f08ba4a30b4b2c4536: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260619 14:32:02.275233 3681 consensus_queue.cc:260] T 9dd56bf4a22d4a7dbccb152b1fde4446 P 76c1a6b944af4ee89210b9c82dffc931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa0316c331a843ba89f6c3258d151bca" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 35249 } } peers { permanent_uuid: "6f74b031fcce43f08ba4a30b4b2c4536" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 42517 } } peers { permanent_uuid: "76c1a6b944af4ee89210b9c82dffc931" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44499 } }
I20260619 14:32:02.275555 3704 ts_tablet_manager.cc:1905] T 464be2061a6947a18a00cdc9e31c6a0d P 6f74b031fcce43f08ba4a30b4b2c4536: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:32:02.277706 3681 ts_tablet_manager.cc:1428] T 9dd56bf4a22d4a7dbccb152b1fde4446 P 76c1a6b944af4ee89210b9c82dffc931: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260619 14:32:02.278524 3704 ts_tablet_manager.cc:1918] T 464be2061a6947a18a00cdc9e31c6a0d P 6f74b031fcce43f08ba4a30b4b2c4536: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:32:02.278647 3704 log.cc:1196] T 464be2061a6947a18a00cdc9e31c6a0d P 6f74b031fcce43f08ba4a30b4b2c4536: Deleting WAL directory at /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/ts-1/wal/wals/464be2061a6947a18a00cdc9e31c6a0d
I20260619 14:32:02.279021 3704 ts_tablet_manager.cc:1939] T 464be2061a6947a18a00cdc9e31c6a0d P 6f74b031fcce43f08ba4a30b4b2c4536: Deleting consensus metadata
I20260619 14:32:02.280166 3161 catalog_manager.cc:4841] TS 6f74b031fcce43f08ba4a30b4b2c4536 (127.2.2.2:42517): tablet 464be2061a6947a18a00cdc9e31c6a0d (table test-table [id=3deb19cb6fd14df5ab05f74ac8ca0ad8]) successfully deleted
I20260619 14:32:02.297935 3688 raft_consensus.cc:482] T 9dd56bf4a22d4a7dbccb152b1fde4446 P 6f74b031fcce43f08ba4a30b4b2c4536 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:32:02.298099 3688 raft_consensus.cc:504] T 9dd56bf4a22d4a7dbccb152b1fde4446 P 6f74b031fcce43f08ba4a30b4b2c4536 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa0316c331a843ba89f6c3258d151bca" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 35249 } } peers { permanent_uuid: "6f74b031fcce43f08ba4a30b4b2c4536" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 42517 } } peers { permanent_uuid: "76c1a6b944af4ee89210b9c82dffc931" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44499 } }
I20260619 14:32:02.299429 3688 leader_election.cc:288] T 9dd56bf4a22d4a7dbccb152b1fde4446 P 6f74b031fcce43f08ba4a30b4b2c4536 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa0316c331a843ba89f6c3258d151bca (127.2.2.1:35249), 76c1a6b944af4ee89210b9c82dffc931 (127.2.2.3:44499)
I20260619 14:32:02.304064 3675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:02.299827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:42115 (local address 127.2.2.1:35249)
0619 14:32:02.299951 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:02.299955 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:02.300204 (+ 249us) server_negotiation.cc:407] Connection header received
0619 14:32:02.300318 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:02.300321 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:02.300378 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:02.300452 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:32:02.300767 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:02.301548 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:02.302989 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:02.303418 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:02.303540 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:02.303617 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:02.303856 (+ 239us) server_negotiation.cc:299] Negotiation successful
0619 14:32:02.303934 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 14:32:02.304095 3709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:02.299634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.1:35249 (local address 127.2.2.2:42115)
0619 14:32:02.300099 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:02.300114 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:02.300225 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:02.300467 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:02.300470 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:02.300483 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:32:02.300604 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:02.300610 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:02.301659 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:02.301662 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:02.302817 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:02.302830 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:02.303546 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:02.303550 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:02.303680 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:02.303706 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 14:32:02.303813 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 14:32:02.303914 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"mutex_wait_us":27,"thread_start_us":84,"threads_started":1}
I20260619 14:32:02.304100 3678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:02.299798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:34903 (local address 127.2.2.3:44499)
0619 14:32:02.299965 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:02.299970 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:02.300255 (+ 285us) server_negotiation.cc:407] Connection header received
0619 14:32:02.300447 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:02.300457 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:02.300505 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:02.300577 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:32:02.300817 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:02.301326 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:02.302952 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:02.303578 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:02.303680 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:02.303742 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:02.303925 (+ 183us) server_negotiation.cc:299] Negotiation successful
0619 14:32:02.303999 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 14:32:02.304479 3710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:02.299634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.3:44499 (local address 127.2.2.2:34903)
0619 14:32:02.300153 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:02.300185 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:02.300324 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:02.300592 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:02.300594 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:02.300607 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:32:02.300721 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:02.300727 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:02.301434 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:02.301439 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:02.302817 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:02.302830 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:02.303684 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:02.303687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:02.303791 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:02.303817 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 14:32:02.304281 (+ 464us) client_negotiation.cc:240] Negotiation successful
0619 14:32:02.304366 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"mutex_wait_us":213,"thread_start_us":46,"threads_started":1}
I20260619 14:32:02.304852 3331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dd56bf4a22d4a7dbccb152b1fde4446" candidate_uuid: "6f74b031fcce43f08ba4a30b4b2c4536" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa0316c331a843ba89f6c3258d151bca" is_pre_election: true
I20260619 14:32:02.305004 3331 raft_consensus.cc:2454] T 9dd56bf4a22d4a7dbccb152b1fde4446 P fa0316c331a843ba89f6c3258d151bca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f74b031fcce43f08ba4a30b4b2c4536 in term 0.
I20260619 14:32:02.305032 3611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dd56bf4a22d4a7dbccb152b1fde4446" candidate_uuid: "6f74b031fcce43f08ba4a30b4b2c4536" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c1a6b944af4ee89210b9c82dffc931" is_pre_election: true
I20260619 14:32:02.305276 3611 raft_consensus.cc:2454] T 9dd56bf4a22d4a7dbccb152b1fde4446 P 76c1a6b944af4ee89210b9c82dffc931 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f74b031fcce43f08ba4a30b4b2c4536 in term 0.
I20260619 14:32:02.305378 3404 leader_election.cc:302] T 9dd56bf4a22d4a7dbccb152b1fde4446 P 6f74b031fcce43f08ba4a30b4b2c4536 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f74b031fcce43f08ba4a30b4b2c4536, fa0316c331a843ba89f6c3258d151bca; no voters:
I20260619 14:32:02.305672 3688 raft_consensus.cc:2781] T 9dd56bf4a22d4a7dbccb152b1fde4446 P 6f74b031fcce43f08ba4a30b4b2c4536 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:32:02.305733 3688 raft_consensus.cc:482] T 9dd56bf4a22d4a7dbccb152b1fde4446 P 6f74b031fcce43f08ba4a30b4b2c4536 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:32:02.305770 3688 raft_consensus.cc:3037] T 9dd56bf4a22d4a7dbccb152b1fde4446 P 6f74b031fcce43f08ba4a30b4b2c4536 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:02.306985 3688 raft_consensus.cc:504] T 9dd56bf4a22d4a7dbccb152b1fde4446 P 6f74b031fcce43f08ba4a30b4b2c4536 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa0316c331a843ba89f6c3258d151bca" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 35249 } } peers { permanent_uuid: "6f74b031fcce43f08ba4a30b4b2c4536" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 42517 } } peers { permanent_uuid: "76c1a6b944af4ee89210b9c82dffc931" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44499 } }
I20260619 14:32:02.307363 3688 leader_election.cc:288] T 9dd56bf4a22d4a7dbccb152b1fde4446 P 6f74b031fcce43f08ba4a30b4b2c4536 [CANDIDATE]: Term 1 election: Requested vote from peers fa0316c331a843ba89f6c3258d151bca (127.2.2.1:35249), 76c1a6b944af4ee89210b9c82dffc931 (127.2.2.3:44499)
I20260619 14:32:02.307693 3611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dd56bf4a22d4a7dbccb152b1fde4446" candidate_uuid: "6f74b031fcce43f08ba4a30b4b2c4536" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c1a6b944af4ee89210b9c82dffc931"
I20260619 14:32:02.307694 3331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dd56bf4a22d4a7dbccb152b1fde4446" candidate_uuid: "6f74b031fcce43f08ba4a30b4b2c4536" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa0316c331a843ba89f6c3258d151bca"
I20260619 14:32:02.307816 3611 raft_consensus.cc:3037] T 9dd56bf4a22d4a7dbccb152b1fde4446 P 76c1a6b944af4ee89210b9c82dffc931 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:02.307816 3331 raft_consensus.cc:3037] T 9dd56bf4a22d4a7dbccb152b1fde4446 P fa0316c331a843ba89f6c3258d151bca [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:02.309387 3331 raft_consensus.cc:2454] T 9dd56bf4a22d4a7dbccb152b1fde4446 P fa0316c331a843ba89f6c3258d151bca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f74b031fcce43f08ba4a30b4b2c4536 in term 1.
I20260619 14:32:02.309485 3611 raft_consensus.cc:2454] T 9dd56bf4a22d4a7dbccb152b1fde4446 P 76c1a6b944af4ee89210b9c82dffc931 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f74b031fcce43f08ba4a30b4b2c4536 in term 1.
I20260619 14:32:02.309765 3403 leader_election.cc:302] T 9dd56bf4a22d4a7dbccb152b1fde4446 P 6f74b031fcce43f08ba4a30b4b2c4536 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f74b031fcce43f08ba4a30b4b2c4536, 76c1a6b944af4ee89210b9c82dffc931; no voters:
I20260619 14:32:02.310039 3688 raft_consensus.cc:2781] T 9dd56bf4a22d4a7dbccb152b1fde4446 P 6f74b031fcce43f08ba4a30b4b2c4536 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:32:02.310285 3688 raft_consensus.cc:686] T 9dd56bf4a22d4a7dbccb152b1fde4446 P 6f74b031fcce43f08ba4a30b4b2c4536 [term 1 LEADER]: Becoming Leader. State: Replica: 6f74b031fcce43f08ba4a30b4b2c4536, State: Running, Role: LEADER
I20260619 14:32:02.310539 3688 consensus_queue.cc:237] T 9dd56bf4a22d4a7dbccb152b1fde4446 P 6f74b031fcce43f08ba4a30b4b2c4536 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa0316c331a843ba89f6c3258d151bca" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 35249 } } peers { permanent_uuid: "6f74b031fcce43f08ba4a30b4b2c4536" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 42517 } } peers { permanent_uuid: "76c1a6b944af4ee89210b9c82dffc931" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44499 } }
I20260619 14:32:02.313009 3172 catalog_manager.cc:5495] T 9dd56bf4a22d4a7dbccb152b1fde4446 P 6f74b031fcce43f08ba4a30b4b2c4536 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f74b031fcce43f08ba4a30b4b2c4536 (127.2.2.2). New cstate: current_term: 1 leader_uuid: "6f74b031fcce43f08ba4a30b4b2c4536" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa0316c331a843ba89f6c3258d151bca" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 35249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f74b031fcce43f08ba4a30b4b2c4536" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 42517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76c1a6b944af4ee89210b9c82dffc931" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44499 } health_report { overall_health: UNKNOWN } } }
Jun 19 14:32:02 dist-test-slave-70rm krb5kdc[3129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879521, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.62@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:32:02.370460 3718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:02.360002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:34767 (local address 127.0.0.1:33044)
0619 14:32:02.360321 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:02.360338 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:02.360483 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:02.360747 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:02.360751 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:02.361282 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:02.361400 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:02.361408 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:02.362173 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:02.362177 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:02.363490 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:02.363501 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:02.363933 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:02.363960 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:02.364049 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:02.364848 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:02.364864 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:02.366666 (+ 1802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:02.368573 (+ 1907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:02.368583 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:02.368590 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:02.368841 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:02.369149 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:02.369155 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:02.369159 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:02.369244 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:02.369749 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:02.369759 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:02.369886 (+ 127us) client_negotiation.cc:769] Sending connection context
0619 14:32:02.370060 (+ 174us) client_negotiation.cc:240] Negotiation successful
0619 14:32:02.370259 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":118,"threads_started":1}
I20260619 14:32:02.370587 3223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:02.360079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33044 (local address 127.2.2.62:34767)
0619 14:32:02.360222 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:02.360227 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:02.360449 (+ 222us) server_negotiation.cc:407] Connection header received
0619 14:32:02.360571 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:02.360574 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:02.360626 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:02.360702 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:02.361528 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:02.362039 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:02.363628 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:02.363842 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:02.363924 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:02.366837 (+ 2913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:02.366841 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:02.366844 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:02.366871 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:02.368466 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:02.368935 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:02.368939 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:02.368940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:02.369029 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:02.369333 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:02.369338 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:02.369341 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:02.369592 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:02.369720 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:02.370168 (+ 448us) server_negotiation.cc:299] Negotiation successful
0619 14:32:02.370385 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 14:32:02.375260 3172 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:33044:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 14:32:02.376156 3172 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:32:02.381682 3311 tablet_service.cc:1467] Processing CreateTablet for tablet c217f55b00f64a8ab25ab53b644c6e7e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ec09f914182549cc8149d7e957818d9c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 14:32:02.381726 3451 tablet_service.cc:1467] Processing CreateTablet for tablet c217f55b00f64a8ab25ab53b644c6e7e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ec09f914182549cc8149d7e957818d9c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 14:32:02.381919 3591 tablet_service.cc:1467] Processing CreateTablet for tablet c217f55b00f64a8ab25ab53b644c6e7e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ec09f914182549cc8149d7e957818d9c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 14:32:02.382171 3451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c217f55b00f64a8ab25ab53b644c6e7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:02.382180 3311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c217f55b00f64a8ab25ab53b644c6e7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:02.382324 3591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c217f55b00f64a8ab25ab53b644c6e7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:02.385285 3681 tablet_bootstrap.cc:492] T c217f55b00f64a8ab25ab53b644c6e7e P 76c1a6b944af4ee89210b9c82dffc931: Bootstrap starting.
I20260619 14:32:02.385376 3683 tablet_bootstrap.cc:492] T c217f55b00f64a8ab25ab53b644c6e7e P 6f74b031fcce43f08ba4a30b4b2c4536: Bootstrap starting.
I20260619 14:32:02.385499 3680 tablet_bootstrap.cc:492] T c217f55b00f64a8ab25ab53b644c6e7e P fa0316c331a843ba89f6c3258d151bca: Bootstrap starting.
I20260619 14:32:02.386379 3681 tablet_bootstrap.cc:654] T c217f55b00f64a8ab25ab53b644c6e7e P 76c1a6b944af4ee89210b9c82dffc931: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:02.386379 3683 tablet_bootstrap.cc:654] T c217f55b00f64a8ab25ab53b644c6e7e P 6f74b031fcce43f08ba4a30b4b2c4536: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:02.386535 3680 tablet_bootstrap.cc:654] T c217f55b00f64a8ab25ab53b644c6e7e P fa0316c331a843ba89f6c3258d151bca: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:02.388254 3680 tablet_bootstrap.cc:492] T c217f55b00f64a8ab25ab53b644c6e7e P fa0316c331a843ba89f6c3258d151bca: No bootstrap required, opened a new log
I20260619 14:32:02.388268 3683 tablet_bootstrap.cc:492] T c217f55b00f64a8ab25ab53b644c6e7e P 6f74b031fcce43f08ba4a30b4b2c4536: No bootstrap required, opened a new log
I20260619 14:32:02.388347 3680 ts_tablet_manager.cc:1397] T c217f55b00f64a8ab25ab53b644c6e7e P fa0316c331a843ba89f6c3258d151bca: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 14:32:02.388350 3683 ts_tablet_manager.cc:1397] T c217f55b00f64a8ab25ab53b644c6e7e P 6f74b031fcce43f08ba4a30b4b2c4536: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 14:32:02.388572 3681 tablet_bootstrap.cc:492] T c217f55b00f64a8ab25ab53b644c6e7e P 76c1a6b944af4ee89210b9c82dffc931: No bootstrap required, opened a new log
I20260619 14:32:02.388654 3681 ts_tablet_manager.cc:1397] T c217f55b00f64a8ab25ab53b644c6e7e P 76c1a6b944af4ee89210b9c82dffc931: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260619 14:32:02.388839 3683 raft_consensus.cc:348] T c217f55b00f64a8ab25ab53b644c6e7e P 6f74b031fcce43f08ba4a30b4b2c4536 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa0316c331a843ba89f6c3258d151bca" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 35249 } } peers { permanent_uuid: "6f74b031fcce43f08ba4a30b4b2c4536" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 42517 } } peers { permanent_uuid: "76c1a6b944af4ee89210b9c82dffc931" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44499 } }
I20260619 14:32:02.388947 3683 raft_consensus.cc:374] T c217f55b00f64a8ab25ab53b644c6e7e P 6f74b031fcce43f08ba4a30b4b2c4536 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:02.388932 3680 raft_consensus.cc:348] T c217f55b00f64a8ab25ab53b644c6e7e P fa0316c331a843ba89f6c3258d151bca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa0316c331a843ba89f6c3258d151bca" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 35249 } } peers { permanent_uuid: "6f74b031fcce43f08ba4a30b4b2c4536" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 42517 } } peers { permanent_uuid: "76c1a6b944af4ee89210b9c82dffc931" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44499 } }
I20260619 14:32:02.388983 3683 raft_consensus.cc:729] T c217f55b00f64a8ab25ab53b644c6e7e P 6f74b031fcce43f08ba4a30b4b2c4536 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f74b031fcce43f08ba4a30b4b2c4536, State: Initialized, Role: FOLLOWER
I20260619 14:32:02.389075 3680 raft_consensus.cc:374] T c217f55b00f64a8ab25ab53b644c6e7e P fa0316c331a843ba89f6c3258d151bca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:02.389120 3680 raft_consensus.cc:729] T c217f55b00f64a8ab25ab53b644c6e7e P fa0316c331a843ba89f6c3258d151bca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa0316c331a843ba89f6c3258d151bca, State: Initialized, Role: FOLLOWER
I20260619 14:32:02.389176 3681 raft_consensus.cc:348] T c217f55b00f64a8ab25ab53b644c6e7e P 76c1a6b944af4ee89210b9c82dffc931 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa0316c331a843ba89f6c3258d151bca" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 35249 } } peers { permanent_uuid: "6f74b031fcce43f08ba4a30b4b2c4536" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 42517 } } peers { permanent_uuid: "76c1a6b944af4ee89210b9c82dffc931" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44499 } }
I20260619 14:32:02.389183 3683 consensus_queue.cc:260] T c217f55b00f64a8ab25ab53b644c6e7e P 6f74b031fcce43f08ba4a30b4b2c4536 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa0316c331a843ba89f6c3258d151bca" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 35249 } } peers { permanent_uuid: "6f74b031fcce43f08ba4a30b4b2c4536" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 42517 } } peers { permanent_uuid: "76c1a6b944af4ee89210b9c82dffc931" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44499 } }
I20260619 14:32:02.389279 3681 raft_consensus.cc:374] T c217f55b00f64a8ab25ab53b644c6e7e P 76c1a6b944af4ee89210b9c82dffc931 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:02.389243 3680 consensus_queue.cc:260] T c217f55b00f64a8ab25ab53b644c6e7e P fa0316c331a843ba89f6c3258d151bca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa0316c331a843ba89f6c3258d151bca" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 35249 } } peers { permanent_uuid: "6f74b031fcce43f08ba4a30b4b2c4536" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 42517 } } peers { permanent_uuid: "76c1a6b944af4ee89210b9c82dffc931" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44499 } }
I20260619 14:32:02.389312 3681 raft_consensus.cc:729] T c217f55b00f64a8ab25ab53b644c6e7e P 76c1a6b944af4ee89210b9c82dffc931 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c1a6b944af4ee89210b9c82dffc931, State: Initialized, Role: FOLLOWER
I20260619 14:32:02.389424 3681 consensus_queue.cc:260] T c217f55b00f64a8ab25ab53b644c6e7e P 76c1a6b944af4ee89210b9c82dffc931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa0316c331a843ba89f6c3258d151bca" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 35249 } } peers { permanent_uuid: "6f74b031fcce43f08ba4a30b4b2c4536" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 42517 } } peers { permanent_uuid: "76c1a6b944af4ee89210b9c82dffc931" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44499 } }
I20260619 14:32:02.389508 3683 ts_tablet_manager.cc:1428] T c217f55b00f64a8ab25ab53b644c6e7e P 6f74b031fcce43f08ba4a30b4b2c4536: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:32:02.389528 3680 ts_tablet_manager.cc:1428] T c217f55b00f64a8ab25ab53b644c6e7e P fa0316c331a843ba89f6c3258d151bca: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:32:02.389678 3681 ts_tablet_manager.cc:1428] T c217f55b00f64a8ab25ab53b644c6e7e P 76c1a6b944af4ee89210b9c82dffc931: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:32:02.389729 3688 tablet_replica.cc:439] T c217f55b00f64a8ab25ab53b644c6e7e P 6f74b031fcce43f08ba4a30b4b2c4536: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa0316c331a843ba89f6c3258d151bca" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 35249 } } peers { permanent_uuid: "6f74b031fcce43f08ba4a30b4b2c4536" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 42517 } } peers { permanent_uuid: "76c1a6b944af4ee89210b9c82dffc931" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44499 } } }
I20260619 14:32:02.389878 3688 tablet_replica.cc:442] T c217f55b00f64a8ab25ab53b644c6e7e P 6f74b031fcce43f08ba4a30b4b2c4536: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:32:02.390014 3698 tablet_replica.cc:439] T c217f55b00f64a8ab25ab53b644c6e7e P 76c1a6b944af4ee89210b9c82dffc931: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa0316c331a843ba89f6c3258d151bca" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 35249 } } peers { permanent_uuid: "6f74b031fcce43f08ba4a30b4b2c4536" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 42517 } } peers { permanent_uuid: "76c1a6b944af4ee89210b9c82dffc931" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44499 } } }
I20260619 14:32:02.390014 3687 tablet_replica.cc:439] T c217f55b00f64a8ab25ab53b644c6e7e P fa0316c331a843ba89f6c3258d151bca: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa0316c331a843ba89f6c3258d151bca" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 35249 } } peers { permanent_uuid: "6f74b031fcce43f08ba4a30b4b2c4536" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 42517 } } peers { permanent_uuid: "76c1a6b944af4ee89210b9c82dffc931" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44499 } } }
I20260619 14:32:02.390196 3687 tablet_replica.cc:442] T c217f55b00f64a8ab25ab53b644c6e7e P fa0316c331a843ba89f6c3258d151bca: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:32:02.391685 3698 tablet_replica.cc:442] T c217f55b00f64a8ab25ab53b644c6e7e P 76c1a6b944af4ee89210b9c82dffc931: This TxnStatusTablet replica's current role is: FOLLOWER
W20260619 14:32:02.506628 3658 tablet.cc:2369] T 9dd56bf4a22d4a7dbccb152b1fde4446 P 76c1a6b944af4ee89210b9c82dffc931: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:32:02.518716 3378 tablet.cc:2369] T 9dd56bf4a22d4a7dbccb152b1fde4446 P fa0316c331a843ba89f6c3258d151bca: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:32:02.519902 3518 tablet.cc:2369] T 9dd56bf4a22d4a7dbccb152b1fde4446 P 6f74b031fcce43f08ba4a30b4b2c4536: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:32:02.566015 3698 raft_consensus.cc:482] T c217f55b00f64a8ab25ab53b644c6e7e P 76c1a6b944af4ee89210b9c82dffc931 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:32:02.566219 3698 raft_consensus.cc:504] T c217f55b00f64a8ab25ab53b644c6e7e P 76c1a6b944af4ee89210b9c82dffc931 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa0316c331a843ba89f6c3258d151bca" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 35249 } } peers { permanent_uuid: "6f74b031fcce43f08ba4a30b4b2c4536" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 42517 } } peers { permanent_uuid: "76c1a6b944af4ee89210b9c82dffc931" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44499 } }
I20260619 14:32:02.566701 3698 leader_election.cc:288] T c217f55b00f64a8ab25ab53b644c6e7e P 76c1a6b944af4ee89210b9c82dffc931 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa0316c331a843ba89f6c3258d151bca (127.2.2.1:35249), 6f74b031fcce43f08ba4a30b4b2c4536 (127.2.2.2:42517)
I20260619 14:32:02.567152 3471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c217f55b00f64a8ab25ab53b644c6e7e" candidate_uuid: "76c1a6b944af4ee89210b9c82dffc931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f74b031fcce43f08ba4a30b4b2c4536" is_pre_election: true
I20260619 14:32:02.567159 3331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c217f55b00f64a8ab25ab53b644c6e7e" candidate_uuid: "76c1a6b944af4ee89210b9c82dffc931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa0316c331a843ba89f6c3258d151bca" is_pre_election: true
I20260619 14:32:02.567314 3471 raft_consensus.cc:2454] T c217f55b00f64a8ab25ab53b644c6e7e P 6f74b031fcce43f08ba4a30b4b2c4536 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c1a6b944af4ee89210b9c82dffc931 in term 0.
I20260619 14:32:02.567323 3331 raft_consensus.cc:2454] T c217f55b00f64a8ab25ab53b644c6e7e P fa0316c331a843ba89f6c3258d151bca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c1a6b944af4ee89210b9c82dffc931 in term 0.
I20260619 14:32:02.567651 3544 leader_election.cc:302] T c217f55b00f64a8ab25ab53b644c6e7e P 76c1a6b944af4ee89210b9c82dffc931 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76c1a6b944af4ee89210b9c82dffc931, fa0316c331a843ba89f6c3258d151bca; no voters:
I20260619 14:32:02.567822 3698 raft_consensus.cc:2781] T c217f55b00f64a8ab25ab53b644c6e7e P 76c1a6b944af4ee89210b9c82dffc931 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:32:02.567878 3698 raft_consensus.cc:482] T c217f55b00f64a8ab25ab53b644c6e7e P 76c1a6b944af4ee89210b9c82dffc931 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:32:02.567905 3698 raft_consensus.cc:3037] T c217f55b00f64a8ab25ab53b644c6e7e P 76c1a6b944af4ee89210b9c82dffc931 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:02.668912 3698 raft_consensus.cc:504] T c217f55b00f64a8ab25ab53b644c6e7e P 76c1a6b944af4ee89210b9c82dffc931 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa0316c331a843ba89f6c3258d151bca" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 35249 } } peers { permanent_uuid: "6f74b031fcce43f08ba4a30b4b2c4536" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 42517 } } peers { permanent_uuid: "76c1a6b944af4ee89210b9c82dffc931" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44499 } }
I20260619 14:32:02.669526 3698 leader_election.cc:288] T c217f55b00f64a8ab25ab53b644c6e7e P 76c1a6b944af4ee89210b9c82dffc931 [CANDIDATE]: Term 1 election: Requested vote from peers fa0316c331a843ba89f6c3258d151bca (127.2.2.1:35249), 6f74b031fcce43f08ba4a30b4b2c4536 (127.2.2.2:42517)
I20260619 14:32:02.670195 3331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c217f55b00f64a8ab25ab53b644c6e7e" candidate_uuid: "76c1a6b944af4ee89210b9c82dffc931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa0316c331a843ba89f6c3258d151bca"
I20260619 14:32:02.670218 3471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c217f55b00f64a8ab25ab53b644c6e7e" candidate_uuid: "76c1a6b944af4ee89210b9c82dffc931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f74b031fcce43f08ba4a30b4b2c4536"
I20260619 14:32:02.670362 3331 raft_consensus.cc:3037] T c217f55b00f64a8ab25ab53b644c6e7e P fa0316c331a843ba89f6c3258d151bca [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:02.670362 3471 raft_consensus.cc:3037] T c217f55b00f64a8ab25ab53b644c6e7e P 6f74b031fcce43f08ba4a30b4b2c4536 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:02.671633 3331 raft_consensus.cc:2454] T c217f55b00f64a8ab25ab53b644c6e7e P fa0316c331a843ba89f6c3258d151bca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c1a6b944af4ee89210b9c82dffc931 in term 1.
I20260619 14:32:02.671633 3471 raft_consensus.cc:2454] T c217f55b00f64a8ab25ab53b644c6e7e P 6f74b031fcce43f08ba4a30b4b2c4536 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c1a6b944af4ee89210b9c82dffc931 in term 1.
I20260619 14:32:02.672039 3545 leader_election.cc:302] T c217f55b00f64a8ab25ab53b644c6e7e P 76c1a6b944af4ee89210b9c82dffc931 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6f74b031fcce43f08ba4a30b4b2c4536, 76c1a6b944af4ee89210b9c82dffc931, fa0316c331a843ba89f6c3258d151bca; no voters:
I20260619 14:32:02.672233 3698 raft_consensus.cc:2781] T c217f55b00f64a8ab25ab53b644c6e7e P 76c1a6b944af4ee89210b9c82dffc931 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:32:02.672320 3698 raft_consensus.cc:686] T c217f55b00f64a8ab25ab53b644c6e7e P 76c1a6b944af4ee89210b9c82dffc931 [term 1 LEADER]: Becoming Leader. State: Replica: 76c1a6b944af4ee89210b9c82dffc931, State: Running, Role: LEADER
I20260619 14:32:02.672485 3698 consensus_queue.cc:237] T c217f55b00f64a8ab25ab53b644c6e7e P 76c1a6b944af4ee89210b9c82dffc931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa0316c331a843ba89f6c3258d151bca" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 35249 } } peers { permanent_uuid: "6f74b031fcce43f08ba4a30b4b2c4536" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 42517 } } peers { permanent_uuid: "76c1a6b944af4ee89210b9c82dffc931" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44499 } }
I20260619 14:32:02.673274 3686 tablet_replica.cc:439] T c217f55b00f64a8ab25ab53b644c6e7e P 76c1a6b944af4ee89210b9c82dffc931: TxnStatusTablet state changed. Reason: New leader 76c1a6b944af4ee89210b9c82dffc931. Latest consensus state: current_term: 1 leader_uuid: "76c1a6b944af4ee89210b9c82dffc931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa0316c331a843ba89f6c3258d151bca" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 35249 } } peers { permanent_uuid: "6f74b031fcce43f08ba4a30b4b2c4536" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 42517 } } peers { permanent_uuid: "76c1a6b944af4ee89210b9c82dffc931" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44499 } } }
I20260619 14:32:02.673382 3686 tablet_replica.cc:442] T c217f55b00f64a8ab25ab53b644c6e7e P 76c1a6b944af4ee89210b9c82dffc931: This TxnStatusTablet replica's current role is: LEADER
I20260619 14:32:02.673684 3723 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 14:32:02.673846 3723 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 14:32:02.674544 3173 catalog_manager.cc:5495] T c217f55b00f64a8ab25ab53b644c6e7e P 76c1a6b944af4ee89210b9c82dffc931 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76c1a6b944af4ee89210b9c82dffc931 (127.2.2.3). New cstate: current_term: 1 leader_uuid: "76c1a6b944af4ee89210b9c82dffc931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa0316c331a843ba89f6c3258d151bca" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 35249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f74b031fcce43f08ba4a30b4b2c4536" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 42517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76c1a6b944af4ee89210b9c82dffc931" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44499 } health_report { overall_health: HEALTHY } } }
I20260619 14:32:02.692054 3718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:02.688347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.3:44499 (local address 127.0.0.1:36344)
0619 14:32:02.688481 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:02.688507 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:02.688607 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:02.688913 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:02.688916 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:02.688926 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:32:02.689052 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:02.689057 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:02.690074 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:02.690076 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:02.690761 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:02.690770 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:02.691327 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:02.691343 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:02.691407 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:02.691455 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:32:02.691766 (+ 311us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:32:02.691785 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 14:32:02.691860 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 14:32:02.691935 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260619 14:32:02.692065 3678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:02.688440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36344 (local address 127.2.2.3:44499)
0619 14:32:02.688576 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:02.688581 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:02.688597 (+ 16us) server_negotiation.cc:407] Connection header received
0619 14:32:02.688718 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:02.688721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:02.688777 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:02.688871 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:32:02.689151 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:02.689937 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:02.690868 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:02.691227 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:02.691327 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:02.691575 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:32:02.691681 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:32:02.691748 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:02.691871 (+ 123us) server_negotiation.cc:299] Negotiation successful
0619 14:32:02.691925 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 14:32:02.694717 3331 raft_consensus.cc:1261] T c217f55b00f64a8ab25ab53b644c6e7e P fa0316c331a843ba89f6c3258d151bca [term 1 FOLLOWER]: Refusing update from remote peer 76c1a6b944af4ee89210b9c82dffc931: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:32:02.694729 3471 raft_consensus.cc:1261] T c217f55b00f64a8ab25ab53b644c6e7e P 6f74b031fcce43f08ba4a30b4b2c4536 [term 1 FOLLOWER]: Refusing update from remote peer 76c1a6b944af4ee89210b9c82dffc931: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:32:02.695197 3698 consensus_queue.cc:1035] T c217f55b00f64a8ab25ab53b644c6e7e P 76c1a6b944af4ee89210b9c82dffc931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa0316c331a843ba89f6c3258d151bca" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 35249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:32:02.695377 3686 consensus_queue.cc:1035] T c217f55b00f64a8ab25ab53b644c6e7e P 76c1a6b944af4ee89210b9c82dffc931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f74b031fcce43f08ba4a30b4b2c4536" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 42517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:32:02.696744 3688 tablet_replica.cc:439] T c217f55b00f64a8ab25ab53b644c6e7e P 6f74b031fcce43f08ba4a30b4b2c4536: TxnStatusTablet state changed. Reason: New leader 76c1a6b944af4ee89210b9c82dffc931. Latest consensus state: current_term: 1 leader_uuid: "76c1a6b944af4ee89210b9c82dffc931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa0316c331a843ba89f6c3258d151bca" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 35249 } } peers { permanent_uuid: "6f74b031fcce43f08ba4a30b4b2c4536" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 42517 } } peers { permanent_uuid: "76c1a6b944af4ee89210b9c82dffc931" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44499 } } }
I20260619 14:32:02.696866 3688 tablet_replica.cc:442] T c217f55b00f64a8ab25ab53b644c6e7e P 6f74b031fcce43f08ba4a30b4b2c4536: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:32:02.697428 3687 tablet_replica.cc:439] T c217f55b00f64a8ab25ab53b644c6e7e P fa0316c331a843ba89f6c3258d151bca: TxnStatusTablet state changed. Reason: New leader 76c1a6b944af4ee89210b9c82dffc931. Latest consensus state: current_term: 1 leader_uuid: "76c1a6b944af4ee89210b9c82dffc931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa0316c331a843ba89f6c3258d151bca" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 35249 } } peers { permanent_uuid: "6f74b031fcce43f08ba4a30b4b2c4536" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 42517 } } peers { permanent_uuid: "76c1a6b944af4ee89210b9c82dffc931" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44499 } } }
I20260619 14:32:02.697525 3687 tablet_replica.cc:442] T c217f55b00f64a8ab25ab53b644c6e7e P fa0316c331a843ba89f6c3258d151bca: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:32:02.699712 3698 tablet_replica.cc:439] T c217f55b00f64a8ab25ab53b644c6e7e P 76c1a6b944af4ee89210b9c82dffc931: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "76c1a6b944af4ee89210b9c82dffc931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa0316c331a843ba89f6c3258d151bca" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 35249 } } peers { permanent_uuid: "6f74b031fcce43f08ba4a30b4b2c4536" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 42517 } } peers { permanent_uuid: "76c1a6b944af4ee89210b9c82dffc931" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44499 } } }
I20260619 14:32:02.699831 3698 tablet_replica.cc:442] T c217f55b00f64a8ab25ab53b644c6e7e P 76c1a6b944af4ee89210b9c82dffc931: This TxnStatusTablet replica's current role is: LEADER
I20260619 14:32:02.701259 3686 tablet_replica.cc:439] T c217f55b00f64a8ab25ab53b644c6e7e P 76c1a6b944af4ee89210b9c82dffc931: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "76c1a6b944af4ee89210b9c82dffc931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa0316c331a843ba89f6c3258d151bca" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 35249 } } peers { permanent_uuid: "6f74b031fcce43f08ba4a30b4b2c4536" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 42517 } } peers { permanent_uuid: "76c1a6b944af4ee89210b9c82dffc931" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44499 } } }
I20260619 14:32:02.701375 3686 tablet_replica.cc:442] T c217f55b00f64a8ab25ab53b644c6e7e P 76c1a6b944af4ee89210b9c82dffc931: This TxnStatusTablet replica's current role is: LEADER
I20260619 14:32:02.701653 3688 tablet_replica.cc:439] T c217f55b00f64a8ab25ab53b644c6e7e P 6f74b031fcce43f08ba4a30b4b2c4536: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "76c1a6b944af4ee89210b9c82dffc931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa0316c331a843ba89f6c3258d151bca" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 35249 } } peers { permanent_uuid: "6f74b031fcce43f08ba4a30b4b2c4536" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 42517 } } peers { permanent_uuid: "76c1a6b944af4ee89210b9c82dffc931" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44499 } } }
I20260619 14:32:02.701759 3688 tablet_replica.cc:442] T c217f55b00f64a8ab25ab53b644c6e7e P 6f74b031fcce43f08ba4a30b4b2c4536: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:32:02.702790 3687 tablet_replica.cc:439] T c217f55b00f64a8ab25ab53b644c6e7e P fa0316c331a843ba89f6c3258d151bca: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "76c1a6b944af4ee89210b9c82dffc931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa0316c331a843ba89f6c3258d151bca" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 35249 } } peers { permanent_uuid: "6f74b031fcce43f08ba4a30b4b2c4536" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 42517 } } peers { permanent_uuid: "76c1a6b944af4ee89210b9c82dffc931" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44499 } } }
I20260619 14:32:02.702911 3687 tablet_replica.cc:442] T c217f55b00f64a8ab25ab53b644c6e7e P fa0316c331a843ba89f6c3258d151bca: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:32:02.710019 3728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:02.705839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48190 (local address 127.2.2.2:42517)
0619 14:32:02.706184 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:02.706192 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:02.706207 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:32:02.706280 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:02.706282 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:02.706334 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:02.706454 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:32:02.706707 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:02.707245 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:02.708318 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:02.708554 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:02.708625 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:02.708974 (+ 349us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:32:02.709562 (+ 588us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:32:02.709692 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:02.709786 (+ 94us) server_negotiation.cc:299] Negotiation successful
0619 14:32:02.709844 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":147,"threads_started":1}
I20260619 14:32:02.721601 3678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:02.718000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:57761 (local address 127.2.2.3:44499)
0619 14:32:02.718172 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:02.718176 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:02.718295 (+ 119us) server_negotiation.cc:407] Connection header received
0619 14:32:02.718420 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:02.718422 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:02.718481 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:02.718547 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:32:02.718788 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:02.719302 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:02.720508 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:02.720733 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:02.720818 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:02.721121 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:32:02.721191 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:32:02.721249 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:02.721361 (+ 112us) server_negotiation.cc:299] Negotiation successful
0619 14:32:02.721411 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 14:32:02.721879 3731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:02.717856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.3:44499 (local address 127.2.2.2:57761)
0619 14:32:02.718221 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:02.718236 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:02.718338 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:02.718572 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:02.718575 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:02.718582 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:32:02.718707 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:02.718713 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:02.719397 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:02.719401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:02.720379 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:02.720389 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:02.720822 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:02.720841 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:02.720939 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:02.720994 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:32:02.721275 (+ 281us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:32:02.721281 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 14:32:02.721344 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 14:32:02.721426 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":86,"threads_started":1}
I20260619 14:32:02.729807 3330 raft_consensus.cc:1261] T 9dd56bf4a22d4a7dbccb152b1fde4446 P fa0316c331a843ba89f6c3258d151bca [term 1 FOLLOWER]: Refusing update from remote peer 6f74b031fcce43f08ba4a30b4b2c4536: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:32:02.729915 3611 raft_consensus.cc:1261] T 9dd56bf4a22d4a7dbccb152b1fde4446 P 76c1a6b944af4ee89210b9c82dffc931 [term 1 FOLLOWER]: Refusing update from remote peer 6f74b031fcce43f08ba4a30b4b2c4536: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:32:02.730347 3711 consensus_queue.cc:1035] T 9dd56bf4a22d4a7dbccb152b1fde4446 P 6f74b031fcce43f08ba4a30b4b2c4536 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa0316c331a843ba89f6c3258d151bca" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 35249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:32:02.730518 3688 consensus_queue.cc:1035] T 9dd56bf4a22d4a7dbccb152b1fde4446 P 6f74b031fcce43f08ba4a30b4b2c4536 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c1a6b944af4ee89210b9c82dffc931" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:32:02.741894 3693 mvcc.cc:205] Tried to move back new op lower bound from 7298578525096218624 to 7298578523386851328. Current Snapshot: MvccSnapshot[applied={T|T < 7298578525096218624}]
I20260619 14:32:02.756671 3728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:02.751918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.3:44273 (local address 127.2.2.2:42517)
0619 14:32:02.752062 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:02.752067 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:02.752338 (+ 271us) server_negotiation.cc:407] Connection header received
0619 14:32:02.752521 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:02.752527 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:02.752593 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:02.752679 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:32:02.752923 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:02.753598 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:02.755134 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:02.755378 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:02.755473 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:02.756061 (+ 588us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:32:02.756163 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:32:02.756287 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:02.756457 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 14:32:02.756526 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 14:32:02.756690 3736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:02.751832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.2:42517 (local address 127.2.2.3:44273)
0619 14:32:02.752219 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:02.752257 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:02.752371 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:02.752692 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:02.752695 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:02.752704 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:32:02.752815 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:02.752822 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:02.753714 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:02.753718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:02.754917 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:02.754932 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:02.755492 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:02.755509 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:02.755713 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:02.755782 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:32:02.756303 (+ 521us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:32:02.756312 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 14:32:02.756415 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 14:32:02.756514 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":129,"threads_started":1}
I20260619 14:32:02.789589 3170 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:32:02.789775 3170 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:32:02.792196 3170 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 25aefb30399c43b7a1516ee1783654fb: Sending DeleteTablet for 3 replicas of tablet 9dd56bf4a22d4a7dbccb152b1fde4446
I20260619 14:32:02.793170 3311 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dd56bf4a22d4a7dbccb152b1fde4446 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:32:02 UTC) from {username='kudu', principal='kudu/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:33980
I20260619 14:32:02.794312 3591 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dd56bf4a22d4a7dbccb152b1fde4446 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:32:02 UTC) from {username='kudu', principal='kudu/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:36328
I20260619 14:32:02.794436 3451 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dd56bf4a22d4a7dbccb152b1fde4446 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:32:02 UTC) from {username='kudu', principal='kudu/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:48182
I20260619 14:32:02.799099 3740 tablet_replica.cc:330] T 9dd56bf4a22d4a7dbccb152b1fde4446 P fa0316c331a843ba89f6c3258d151bca: stopping tablet replica
I20260619 14:32:02.799288 3740 raft_consensus.cc:2229] T 9dd56bf4a22d4a7dbccb152b1fde4446 P fa0316c331a843ba89f6c3258d151bca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:32:02.799456 3740 raft_consensus.cc:2258] T 9dd56bf4a22d4a7dbccb152b1fde4446 P fa0316c331a843ba89f6c3258d151bca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:32:02.800148 3740 ts_tablet_manager.cc:1905] T 9dd56bf4a22d4a7dbccb152b1fde4446 P fa0316c331a843ba89f6c3258d151bca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:32:02.802305 3742 tablet_replica.cc:330] T 9dd56bf4a22d4a7dbccb152b1fde4446 P 76c1a6b944af4ee89210b9c82dffc931: stopping tablet replica
I20260619 14:32:02.802450 3742 raft_consensus.cc:2229] T 9dd56bf4a22d4a7dbccb152b1fde4446 P 76c1a6b944af4ee89210b9c82dffc931 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:32:02.802603 3742 raft_consensus.cc:2258] T 9dd56bf4a22d4a7dbccb152b1fde4446 P 76c1a6b944af4ee89210b9c82dffc931 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:32:02.802685 3743 tablet_replica.cc:330] T 9dd56bf4a22d4a7dbccb152b1fde4446 P 6f74b031fcce43f08ba4a30b4b2c4536: stopping tablet replica
I20260619 14:32:02.802829 3743 raft_consensus.cc:2229] T 9dd56bf4a22d4a7dbccb152b1fde4446 P 6f74b031fcce43f08ba4a30b4b2c4536 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:32:02.803154 3743 raft_consensus.cc:2258] T 9dd56bf4a22d4a7dbccb152b1fde4446 P 6f74b031fcce43f08ba4a30b4b2c4536 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:32:02.804353 3743 ts_tablet_manager.cc:1905] T 9dd56bf4a22d4a7dbccb152b1fde4446 P 6f74b031fcce43f08ba4a30b4b2c4536: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:32:02.806818 3742 ts_tablet_manager.cc:1905] T 9dd56bf4a22d4a7dbccb152b1fde4446 P 76c1a6b944af4ee89210b9c82dffc931: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 14:32:02 dist-test-slave-70rm krb5kdc[3129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879522, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.2.1@KRBTEST.COM
I20260619 14:32:02.815205 3675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:02.798708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33986 (local address 127.2.2.1:35249)
0619 14:32:02.798855 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:02.798860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:02.803649 (+ 4789us) server_negotiation.cc:407] Connection header received
0619 14:32:02.803824 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:02.803828 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:02.803885 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:02.803964 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:02.805267 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:02.806025 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:02.807292 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:02.807626 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:02.807709 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:02.811361 (+ 3652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:02.811377 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:02.811399 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:02.811429 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:02.813248 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:02.813752 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:02.813758 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:02.813759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:02.813852 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:02.814187 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:02.814190 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:02.814192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:02.814464 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:02.814711 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:02.814891 (+ 180us) server_negotiation.cc:299] Negotiation successful
0619 14:32:02.815022 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260619 14:32:02.815850 3371 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33986
I20260619 14:32:02.825769 3223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:02.817545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33060 (local address 127.2.2.62:34767)
0619 14:32:02.817692 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:02.817696 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:02.818080 (+ 384us) server_negotiation.cc:407] Connection header received
0619 14:32:02.818236 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:02.818239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:02.818294 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:02.818382 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:02.819039 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:02.819581 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:02.820473 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:02.820726 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:02.820839 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:02.822374 (+ 1535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:02.822378 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:02.822381 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:02.822417 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:02.824049 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:02.824493 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:02.824497 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:02.824499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:02.824594 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:02.824889 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:02.824894 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:02.824896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:02.825094 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:02.825227 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:02.825474 (+ 247us) server_negotiation.cc:299] Negotiation successful
0619 14:32:02.825602 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260619 14:32:02.826539 3170 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33060
I20260619 14:32:02.827931 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 3246
W20260619 14:32:02.872916 3160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.2.1:35249 (error 108)
I20260619 14:32:02.873131 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 3386
I20260619 14:32:02.875165 3748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:02.874768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.1:35249 (local address 127.0.0.1:33990)
0619 14:32:02.875002 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:02.875072 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.1:35249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":55,"threads_started":1}
W20260619 14:32:02.875325 3160 catalog_manager.cc:4584] TS fa0316c331a843ba89f6c3258d151bca (127.2.2.1:35249): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9dd56bf4a22d4a7dbccb152b1fde4446: Network error: Client connection negotiation failed: client connection to 127.2.2.1:35249: connect: Connection refused (error 111)
I20260619 14:32:02.928649 3748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:02.928236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.1:35249 (local address 127.0.0.1:33992)
0619 14:32:02.928404 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:02.928474 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.1:35249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260619 14:32:02.929446 3748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:02.929213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.1:35249 (local address 127.0.0.1:33996)
0619 14:32:02.929336 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:02.929387 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.1:35249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 14:32:02.964171 3742 ts_tablet_manager.cc:1918] T 9dd56bf4a22d4a7dbccb152b1fde4446 P 76c1a6b944af4ee89210b9c82dffc931: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:32:02.964316 3742 log.cc:1196] T 9dd56bf4a22d4a7dbccb152b1fde4446 P 76c1a6b944af4ee89210b9c82dffc931: Deleting WAL directory at /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879514993456-2056-0/minicluster-data/ts-2/wal/wals/9dd56bf4a22d4a7dbccb152b1fde4446
I20260619 14:32:02.964702 3742 ts_tablet_manager.cc:1939] T 9dd56bf4a22d4a7dbccb152b1fde4446 P 76c1a6b944af4ee89210b9c82dffc931: Deleting consensus metadata
I20260619 14:32:02.965713 3159 catalog_manager.cc:4841] TS 76c1a6b944af4ee89210b9c82dffc931 (127.2.2.3:44499): tablet 9dd56bf4a22d4a7dbccb152b1fde4446 (table test-table [id=db3196998e5e43178ace6c34ad883450]) successfully deleted
I20260619 14:32:02.969012 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 3526
I20260619 14:32:02.969939 3748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:02.969253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.2:42517 (local address 127.0.0.1:48198)
0619 14:32:02.969765 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:02.969831 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.2:42517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412}
I20260619 14:32:02.976183 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 3142
2026-06-19T14:32:02Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3130 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:32:05 dist-test-slave-70rm krb5kdc[3751](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:32:05 dist-test-slave-70rm krb5kdc[3751](info): set up 2 sockets
Jun 19 14:32:05 dist-test-slave-70rm krb5kdc[3751](info): commencing operation
krb5kdc: starting...
W20260619 14:32:06.573998 2056 mini_kdc.cc:121] Time spent starting KDC: real 3.574s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Jun 19 14:32:06 dist-test-slave-70rm krb5kdc[3751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:32:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:32:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.2.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab.
Entry for principal kudu/127.2.2.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab.
I20260619 14:32:06.704936 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:42985
--webserver_interface=127.2.2.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:45941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.2.62:42985
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:32:06.815546 3764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:06.815832 3764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:06.815873 3764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:06.820135 3764 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:32:06.820202 3764 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 14:32:06.820219 3764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:06.820238 3764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:06.820277 3764 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:32:06.820312 3764 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:32:06.825673 3764 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:45941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.2.62:42985
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:42985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.2.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:32:06.826858 3764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:06.827930 3764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:32:06.836580 3769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:06.836606 3770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:06.836637 3772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:06.836812 3764 server_base.cc:1034] running on GCE node
I20260619 14:32:06.837523 3764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:06.838806 3764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:06.840010 3764 hybrid_clock.cc:648] HybridClock initialized: now 1781879526839997 us; error 34 us; skew 500 ppm
Jun 19 14:32:06 dist-test-slave-70rm krb5kdc[3751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879526, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:32:06.848327 3764 init.cc:377] Logged in from keytab as kudu/127.2.2.62@KRBTEST.COM (short username kudu)
I20260619 14:32:06.849380 3764 webserver.cc:460] Webserver started at http://127.2.2.62:41705/ using document root <none> and password file <none>
I20260619 14:32:06.849985 3764 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:06.850080 3764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:06.850347 3764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:32:06.852523 3764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/master-0/data/instance:
uuid: "de380797462143cc92218c66aee9846e"
format_stamp: "Formatted at 2026-06-19 14:32:06 on dist-test-slave-70rm"
server_key: "202cfe4881fea873cda431caa3e429ee"
server_key_iv: "703ee2c95a7e3b720b59a9eefe595dea"
server_key_version: "encryptionkey@0"
I20260619 14:32:06.853121 3764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/master-0/wal/instance:
uuid: "de380797462143cc92218c66aee9846e"
format_stamp: "Formatted at 2026-06-19 14:32:06 on dist-test-slave-70rm"
server_key: "202cfe4881fea873cda431caa3e429ee"
server_key_iv: "703ee2c95a7e3b720b59a9eefe595dea"
server_key_version: "encryptionkey@0"
I20260619 14:32:06.857204 3764 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 14:32:06.859839 3779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:06.861112 3764 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:32:06.861267 3764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/master-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/master-0/wal
uuid: "de380797462143cc92218c66aee9846e"
format_stamp: "Formatted at 2026-06-19 14:32:06 on dist-test-slave-70rm"
server_key: "202cfe4881fea873cda431caa3e429ee"
server_key_iv: "703ee2c95a7e3b720b59a9eefe595dea"
server_key_version: "encryptionkey@0"
I20260619 14:32:06.861389 3764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:06.880751 3764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:06.884284 3764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:32:06.884533 3764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:06.893433 3764 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.62:42985
I20260619 14:32:06.893441 3841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.62:42985 every 8 connection(s)
I20260619 14:32:06.894569 3764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
I20260619 14:32:06.897531 3842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:06.901566 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 3764
I20260619 14:32:06.901702 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/master-0/wal/instance
I20260619 14:32:06.901958 2056 external_mini_cluster.cc:1428] Setting key 0a06d462abd48259e78e1be089ce03c4
I20260619 14:32:06.904203 3842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de380797462143cc92218c66aee9846e: Bootstrap starting.
I20260619 14:32:06.906822 3842 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de380797462143cc92218c66aee9846e: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:06.907661 3842 log.cc:824] T 00000000000000000000000000000000 P de380797462143cc92218c66aee9846e: Log is configured to *not* fsync() on all Append() calls
I20260619 14:32:06.909970 3842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de380797462143cc92218c66aee9846e: No bootstrap required, opened a new log
Jun 19 14:32:06 dist-test-slave-70rm krb5kdc[3751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:32:06.914083 3842 raft_consensus.cc:348] T 00000000000000000000000000000000 P de380797462143cc92218c66aee9846e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de380797462143cc92218c66aee9846e" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 42985 } }
I20260619 14:32:06.914397 3842 raft_consensus.cc:374] T 00000000000000000000000000000000 P de380797462143cc92218c66aee9846e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:06.914462 3842 raft_consensus.cc:729] T 00000000000000000000000000000000 P de380797462143cc92218c66aee9846e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de380797462143cc92218c66aee9846e, State: Initialized, Role: FOLLOWER
I20260619 14:32:06.914983 3842 consensus_queue.cc:260] T 00000000000000000000000000000000 P de380797462143cc92218c66aee9846e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de380797462143cc92218c66aee9846e" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 42985 } }
I20260619 14:32:06.915194 3842 raft_consensus.cc:388] T 00000000000000000000000000000000 P de380797462143cc92218c66aee9846e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:32:06.915267 3842 raft_consensus.cc:482] T 00000000000000000000000000000000 P de380797462143cc92218c66aee9846e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:32:06.915366 3842 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de380797462143cc92218c66aee9846e [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:06.916628 3842 raft_consensus.cc:504] T 00000000000000000000000000000000 P de380797462143cc92218c66aee9846e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de380797462143cc92218c66aee9846e" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 42985 } }
I20260619 14:32:06.916999 3842 leader_election.cc:302] T 00000000000000000000000000000000 P de380797462143cc92218c66aee9846e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de380797462143cc92218c66aee9846e; no voters:
I20260619 14:32:06.917361 3842 leader_election.cc:288] T 00000000000000000000000000000000 P de380797462143cc92218c66aee9846e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:32:06.917479 3847 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de380797462143cc92218c66aee9846e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:32:06.917749 3847 raft_consensus.cc:686] T 00000000000000000000000000000000 P de380797462143cc92218c66aee9846e [term 1 LEADER]: Becoming Leader. State: Replica: de380797462143cc92218c66aee9846e, State: Running, Role: LEADER
I20260619 14:32:06.917840 3845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:06.903466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59810 (local address 127.2.2.62:42985)
0619 14:32:06.903897 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:06.903914 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:06.903950 (+ 36us) server_negotiation.cc:407] Connection header received
0619 14:32:06.904658 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:06.904693 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:06.905048 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:06.905347 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:06.906268 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:06.907357 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:06.908475 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:06.908961 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:06.909086 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:06.912124 (+ 3038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:06.912129 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:06.912143 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:06.912174 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:06.914275 (+ 2101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:06.914834 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:06.914840 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:06.914846 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:06.914993 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:06.915379 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:06.915383 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:06.915385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:06.915930 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:06.916135 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:06.916462 (+ 327us) server_negotiation.cc:299] Negotiation successful
0619 14:32:06.916804 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":149,"threads_started":1}
I20260619 14:32:06.918100 3847 consensus_queue.cc:237] T 00000000000000000000000000000000 P de380797462143cc92218c66aee9846e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de380797462143cc92218c66aee9846e" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 42985 } }
I20260619 14:32:06.918669 3842 sys_catalog.cc:564] T 00000000000000000000000000000000 P de380797462143cc92218c66aee9846e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:32:06.920804 3849 sys_catalog.cc:455] T 00000000000000000000000000000000 P de380797462143cc92218c66aee9846e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de380797462143cc92218c66aee9846e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de380797462143cc92218c66aee9846e" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 42985 } } }
I20260619 14:32:06.921012 3848 sys_catalog.cc:455] T 00000000000000000000000000000000 P de380797462143cc92218c66aee9846e [sys.catalog]: SysCatalogTable state changed. Reason: New leader de380797462143cc92218c66aee9846e. Latest consensus state: current_term: 1 leader_uuid: "de380797462143cc92218c66aee9846e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de380797462143cc92218c66aee9846e" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 42985 } } }
I20260619 14:32:06.921244 3849 sys_catalog.cc:458] T 00000000000000000000000000000000 P de380797462143cc92218c66aee9846e [sys.catalog]: This master's current role is: LEADER
I20260619 14:32:06.921245 3848 sys_catalog.cc:458] T 00000000000000000000000000000000 P de380797462143cc92218c66aee9846e [sys.catalog]: This master's current role is: LEADER
W20260619 14:32:06.923879 3861 catalog_manager.cc:1558] T 00000000000000000000000000000000 P de380797462143cc92218c66aee9846e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:32:06.923964 3861 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:32:06.924060 3863 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:32:06.924793 3863 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:32:06.930518 3863 catalog_manager.cc:1347] Generated new cluster ID: 9db36c4e202c4c0385ab402dea19401f
I20260619 14:32:06.930600 3863 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:32:06.943770 3863 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:32:06.944793 3863 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:32:06.957412 3863 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de380797462143cc92218c66aee9846e: Generated new TSK 0
I20260619 14:32:06.958292 3863 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.2.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab.
Entry for principal kudu/127.2.2.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab.
I20260619 14:32:06.989233 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.1:0
--local_ip_for_outbound_sockets=127.2.2.1
--webserver_interface=127.2.2.1
--webserver_port=0
--tserver_master_addrs=127.2.2.62:42985
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab
--principal=kudu/127.2.2.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:45941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:32:07.100519 3868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:07.100786 3868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:07.100832 3868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:07.104872 3868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:07.104940 3868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:07.104980 3868 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 14:32:07.104995 3868 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 14:32:07.105027 3868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.1
I20260619 14:32:07.110450 3868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:45941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab
--principal=kudu/127.2.2.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.2.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.2.62:42985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.1
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:32:07.111568 3868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:07.112569 3868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:32:07.122655 3873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:07.122766 3874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:07.122881 3868 server_base.cc:1034] running on GCE node
W20260619 14:32:07.122670 3876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:07.123458 3868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:07.124145 3868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:07.125310 3868 hybrid_clock.cc:648] HybridClock initialized: now 1781879527125271 us; error 52 us; skew 500 ppm
Jun 19 14:32:07 dist-test-slave-70rm krb5kdc[3751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879527, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:32:07.128259 3868 init.cc:377] Logged in from keytab as kudu/127.2.2.1@KRBTEST.COM (short username kudu)
I20260619 14:32:07.129163 3868 webserver.cc:460] Webserver started at http://127.2.2.1:35575/ using document root <none> and password file <none>
I20260619 14:32:07.129757 3868 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:07.129849 3868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:07.130085 3868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:32:07.132225 3868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/ts-0/data/instance:
uuid: "5bb5a7292e2248d28331461c809f2971"
format_stamp: "Formatted at 2026-06-19 14:32:07 on dist-test-slave-70rm"
server_key: "328ce38b45b6872fa716983aa16fadd0"
server_key_iv: "4dff11d264799e29664721f220a18253"
server_key_version: "encryptionkey@0"
I20260619 14:32:07.132823 3868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/ts-0/wal/instance:
uuid: "5bb5a7292e2248d28331461c809f2971"
format_stamp: "Formatted at 2026-06-19 14:32:07 on dist-test-slave-70rm"
server_key: "328ce38b45b6872fa716983aa16fadd0"
server_key_iv: "4dff11d264799e29664721f220a18253"
server_key_version: "encryptionkey@0"
I20260619 14:32:07.136870 3868 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260619 14:32:07.139361 3883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:07.140684 3868 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:32:07.140829 3868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/ts-0/wal
uuid: "5bb5a7292e2248d28331461c809f2971"
format_stamp: "Formatted at 2026-06-19 14:32:07 on dist-test-slave-70rm"
server_key: "328ce38b45b6872fa716983aa16fadd0"
server_key_iv: "4dff11d264799e29664721f220a18253"
server_key_version: "encryptionkey@0"
I20260619 14:32:07.140954 3868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:07.164791 3868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:07.168179 3868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:32:07.168375 3868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:07.169904 3868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:32:07.169956 3868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:07.169997 3868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:32:07.170053 3868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 14:32:07 dist-test-slave-70rm krb5kdc[3751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879527, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.1@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:32:07.180872 3868 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.1:42881
I20260619 14:32:07.180979 3998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.1:42881 every 8 connection(s)
I20260619 14:32:07.182066 3868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
I20260619 14:32:07.184199 3845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:07.170956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:40727 (local address 127.2.2.62:42985)
0619 14:32:07.171109 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:07.171113 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:07.172184 (+ 1071us) server_negotiation.cc:407] Connection header received
0619 14:32:07.173345 (+ 1161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:07.173350 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:07.173418 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:07.173519 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:07.175001 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:07.175896 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:07.177270 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:07.177506 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:07.177628 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:07.180289 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:07.180293 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:07.180296 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:07.180326 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:07.182195 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:07.182793 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:07.182797 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:07.182799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:07.182892 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:07.183264 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:07.183267 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:07.183268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:07.183445 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:07.183523 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:07.183968 (+ 445us) server_negotiation.cc:299] Negotiation successful
0619 14:32:07.184079 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 14:32:07.185012 3892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:07.171547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:42985 (local address 127.2.2.1:40727)
0619 14:32:07.172028 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:07.172074 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:07.173084 (+ 1010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:07.173792 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:07.173799 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:07.174323 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:07.174847 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:07.174862 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:07.176049 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:07.176054 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:07.177139 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:07.177149 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:07.177613 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:07.177669 (+ 56us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:07.177855 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:07.178496 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:07.178513 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:07.180106 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:07.182307 (+ 2201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:07.182311 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:07.182323 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:07.182685 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:07.182993 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:07.182995 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:07.182996 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:07.183194 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:07.183542 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:07.183546 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:07.183766 (+ 220us) client_negotiation.cc:769] Sending connection context
0619 14:32:07.183918 (+ 152us) client_negotiation.cc:240] Negotiation successful
0619 14:32:07.184167 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":123,"threads_started":1}
I20260619 14:32:07.185912 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 3868
I20260619 14:32:07.186053 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/ts-0/wal/instance
I20260619 14:32:07.186410 2056 external_mini_cluster.cc:1428] Setting key 18a6c9a16f9cad058d3cb2108b4587fa
I20260619 14:32:07.186405 3999 heartbeater.cc:347] Connected to a master server at 127.2.2.62:42985
I20260619 14:32:07.186549 3999 heartbeater.cc:464] Registering TS with master...
I20260619 14:32:07.187126 3999 heartbeater.cc:511] Master 127.2.2.62:42985 requested a full tablet report, sending...
I20260619 14:32:07.188712 3795 ts_manager.cc:194] Registered new tserver with Master: 5bb5a7292e2248d28331461c809f2971 (127.2.2.1:42881)
I20260619 14:32:07.190179 3795 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.2.1@KRBTEST.COM'} at 127.2.2.1:40727
WARNING: no policy specified for kudu/127.2.2.2@KRBTEST.COM; defaulting to no policy
I20260619 14:32:07.196308 3845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:07.188927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:56129 (local address 127.2.2.62:42985)
0619 14:32:07.189031 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:07.189035 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:07.189066 (+ 31us) server_negotiation.cc:407] Connection header received
0619 14:32:07.189189 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:07.189192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:07.189231 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:07.189306 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:07.189963 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:07.190555 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:07.191608 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:07.191838 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:07.191946 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:07.193109 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:07.193112 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:07.193115 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:07.193138 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:07.194677 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:07.195111 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:07.195114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:07.195116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:07.195198 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:07.195484 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:07.195487 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:07.195489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:07.195671 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:07.195776 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:07.196041 (+ 265us) server_negotiation.cc:299] Negotiation successful
0619 14:32:07.196154 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 14:32:07.196321 4005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:07.188679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:42985 (local address 127.2.2.1:56129)
0619 14:32:07.188987 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:07.189005 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:07.189088 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:07.189351 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:07.189357 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:07.189624 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:07.189780 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:07.189791 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:07.190682 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:07.190685 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:07.191473 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:07.191482 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:07.191935 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:07.191953 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:07.192088 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:07.192624 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:07.192636 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:07.192964 (+ 328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:07.194790 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:07.194794 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:07.194797 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:07.194996 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:07.195289 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:07.195293 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:07.195296 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:07.195384 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:07.195764 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:07.195767 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:07.195866 (+ 99us) client_negotiation.cc:769] Sending connection context
0619 14:32:07.195971 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 14:32:07.196116 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":29,"thread_start_us":92,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab.
Entry for principal kudu/127.2.2.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab.
I20260619 14:32:07.213258 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.2:0
--local_ip_for_outbound_sockets=127.2.2.2
--webserver_interface=127.2.2.2
--webserver_port=0
--tserver_master_addrs=127.2.2.62:42985
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab
--principal=kudu/127.2.2.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:45941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:32:07.325526 4008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:07.325836 4008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:07.325927 4008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:07.330051 4008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:07.330159 4008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:07.330235 4008 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 14:32:07.330279 4008 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 14:32:07.330348 4008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.2
I20260619 14:32:07.335783 4008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:45941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab
--principal=kudu/127.2.2.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.2.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.2.62:42985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.2
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:32:07.336879 4008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:07.337913 4008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:32:07.347565 4014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:07.347568 4013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:07.347638 4016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:07.347723 4008 server_base.cc:1034] running on GCE node
I20260619 14:32:07.348459 4008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:07.349123 4008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:07.350278 4008 hybrid_clock.cc:648] HybridClock initialized: now 1781879527350249 us; error 47 us; skew 500 ppm
Jun 19 14:32:07 dist-test-slave-70rm krb5kdc[3751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879527, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:32:07.353048 4008 init.cc:377] Logged in from keytab as kudu/127.2.2.2@KRBTEST.COM (short username kudu)
I20260619 14:32:07.354009 4008 webserver.cc:460] Webserver started at http://127.2.2.2:37087/ using document root <none> and password file <none>
I20260619 14:32:07.354740 4008 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:07.354808 4008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:07.355046 4008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:32:07.357151 4008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/ts-1/data/instance:
uuid: "ff059b5728f94491b9e2d7c8a97c641e"
format_stamp: "Formatted at 2026-06-19 14:32:07 on dist-test-slave-70rm"
server_key: "211187824bc9111226677625589ebad1"
server_key_iv: "e81debfcab67e1b8b1de8d5b306bae65"
server_key_version: "encryptionkey@0"
I20260619 14:32:07.357740 4008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/ts-1/wal/instance:
uuid: "ff059b5728f94491b9e2d7c8a97c641e"
format_stamp: "Formatted at 2026-06-19 14:32:07 on dist-test-slave-70rm"
server_key: "211187824bc9111226677625589ebad1"
server_key_iv: "e81debfcab67e1b8b1de8d5b306bae65"
server_key_version: "encryptionkey@0"
I20260619 14:32:07.361541 4008 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 14:32:07.364063 4023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:07.365535 4008 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:32:07.365669 4008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/ts-1/wal
uuid: "ff059b5728f94491b9e2d7c8a97c641e"
format_stamp: "Formatted at 2026-06-19 14:32:07 on dist-test-slave-70rm"
server_key: "211187824bc9111226677625589ebad1"
server_key_iv: "e81debfcab67e1b8b1de8d5b306bae65"
server_key_version: "encryptionkey@0"
I20260619 14:32:07.365784 4008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:07.379215 4008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:07.382678 4008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:32:07.382853 4008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:07.384387 4008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:32:07.384443 4008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:07.384524 4008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:32:07.384560 4008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 14:32:07 dist-test-slave-70rm krb5kdc[3751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879527, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.2@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:32:07.397116 4008 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.2:38205
I20260619 14:32:07.397432 4138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.2:38205 every 8 connection(s)
I20260619 14:32:07.398406 4008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
I20260619 14:32:07.399744 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 4008
I20260619 14:32:07.399839 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/ts-1/wal/instance
I20260619 14:32:07.400174 2056 external_mini_cluster.cc:1428] Setting key 0b3bada861e33b380c4d5c0f72b490fb
I20260619 14:32:07.400918 3845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:07.385458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:47617 (local address 127.2.2.62:42985)
0619 14:32:07.385604 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:07.385607 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:07.386911 (+ 1304us) server_negotiation.cc:407] Connection header received
0619 14:32:07.388344 (+ 1433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:07.388347 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:07.388407 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:07.388502 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:07.390570 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:07.391438 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:07.392768 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:07.393136 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:07.393277 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:07.396670 (+ 3393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:07.396674 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:07.396677 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:07.396714 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:07.398432 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:07.399025 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:07.399028 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:07.399030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:07.399131 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:07.399534 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:07.399538 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:07.399540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:07.399774 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:07.399915 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:07.400614 (+ 699us) server_negotiation.cc:299] Negotiation successful
0619 14:32:07.400764 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 14:32:07.401835 4032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:07.386066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:42985 (local address 127.2.2.2:47617)
0619 14:32:07.386690 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:07.386787 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:07.388086 (+ 1299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:07.388747 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:07.388759 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:07.389382 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:07.390344 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:07.390372 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:07.391593 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:07.391596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:07.392626 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:07.392636 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:07.393278 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:07.393356 (+ 78us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:07.393612 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:07.394352 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:07.394380 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:07.396470 (+ 2090us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:07.398591 (+ 2121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:07.398597 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:07.398611 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:07.398894 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:07.399235 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:07.399239 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:07.399243 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:07.399407 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:07.399948 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:07.399955 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:07.400288 (+ 333us) client_negotiation.cc:769] Sending connection context
0619 14:32:07.400506 (+ 218us) client_negotiation.cc:240] Negotiation successful
0619 14:32:07.400865 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":146,"threads_started":1}
I20260619 14:32:07.403245 4139 heartbeater.cc:347] Connected to a master server at 127.2.2.62:42985
I20260619 14:32:07.403383 4139 heartbeater.cc:464] Registering TS with master...
I20260619 14:32:07.403924 4139 heartbeater.cc:511] Master 127.2.2.62:42985 requested a full tablet report, sending...
I20260619 14:32:07.405015 3796 ts_manager.cc:194] Registered new tserver with Master: ff059b5728f94491b9e2d7c8a97c641e (127.2.2.2:38205)
I20260619 14:32:07.405798 3796 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.2.2@KRBTEST.COM'} at 127.2.2.2:47617
WARNING: no policy specified for kudu/127.2.2.3@KRBTEST.COM; defaulting to no policy
I20260619 14:32:07.414048 3845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:07.405601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:39157 (local address 127.2.2.62:42985)
0619 14:32:07.406162 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:07.406169 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:07.406420 (+ 251us) server_negotiation.cc:407] Connection header received
0619 14:32:07.406550 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:07.406555 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:07.406610 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:07.406689 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:07.407185 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:07.408046 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:07.408855 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:07.409280 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:07.409415 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:07.410603 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:07.410607 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:07.410612 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:07.410642 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:07.412412 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:07.412940 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:07.412946 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:07.412951 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:07.413055 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:07.413298 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:07.413300 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:07.413301 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:07.413510 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:07.413584 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:07.413825 (+ 241us) server_negotiation.cc:299] Negotiation successful
0619 14:32:07.413923 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459}
I20260619 14:32:07.414155 4146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:07.405517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:42985 (local address 127.2.2.2:39157)
0619 14:32:07.406327 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:07.406342 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:07.406444 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:07.406726 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:07.406729 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:07.406946 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:07.407057 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:07.407064 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:07.408176 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:07.408178 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:07.408740 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:07.408749 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:07.409385 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:07.409401 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:07.409468 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:07.410050 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:07.410061 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:07.410469 (+ 408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:07.412559 (+ 2090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:07.412562 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:07.412563 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:07.412823 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:07.413148 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:07.413150 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:07.413151 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:07.413229 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:07.413587 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:07.413589 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:07.413667 (+ 78us) client_negotiation.cc:769] Sending connection context
0619 14:32:07.413746 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 14:32:07.413939 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":724,"thread_start_us":88,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab.
Entry for principal kudu/127.2.2.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab.
I20260619 14:32:07.433816 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.3:0
--local_ip_for_outbound_sockets=127.2.2.3
--webserver_interface=127.2.2.3
--webserver_port=0
--tserver_master_addrs=127.2.2.62:42985
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab
--principal=kudu/127.2.2.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:45941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:32:07.546620 4148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:07.546882 4148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:07.546962 4148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:07.551165 4148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:07.551240 4148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:07.551281 4148 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 14:32:07.551295 4148 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 14:32:07.551369 4148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.3
I20260619 14:32:07.556847 4148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:45941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab
--principal=kudu/127.2.2.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.2.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.2.62:42985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.3
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:32:07.557956 4148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:07.559093 4148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:32:07.569037 4154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:07.569031 4156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:07.569037 4153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:07.569075 4148 server_base.cc:1034] running on GCE node
I20260619 14:32:07.569738 4148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:07.570506 4148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:07.571668 4148 hybrid_clock.cc:648] HybridClock initialized: now 1781879527571633 us; error 51 us; skew 500 ppm
Jun 19 14:32:07 dist-test-slave-70rm krb5kdc[3751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879527, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:32:07.574431 4148 init.cc:377] Logged in from keytab as kudu/127.2.2.3@KRBTEST.COM (short username kudu)
I20260619 14:32:07.575256 4148 webserver.cc:460] Webserver started at http://127.2.2.3:43983/ using document root <none> and password file <none>
I20260619 14:32:07.575767 4148 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:07.575824 4148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:07.575985 4148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:32:07.577960 4148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/ts-2/data/instance:
uuid: "391bb2553f8a44ee837bcd431341b7e5"
format_stamp: "Formatted at 2026-06-19 14:32:07 on dist-test-slave-70rm"
server_key: "c49d49f329526e4f348f62aeb25f367e"
server_key_iv: "69fa5c3936e1e67febf5fab9330ae868"
server_key_version: "encryptionkey@0"
I20260619 14:32:07.578547 4148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/ts-2/wal/instance:
uuid: "391bb2553f8a44ee837bcd431341b7e5"
format_stamp: "Formatted at 2026-06-19 14:32:07 on dist-test-slave-70rm"
server_key: "c49d49f329526e4f348f62aeb25f367e"
server_key_iv: "69fa5c3936e1e67febf5fab9330ae868"
server_key_version: "encryptionkey@0"
I20260619 14:32:07.583254 4148 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:32:07.586588 4163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:07.588410 4148 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:32:07.588551 4148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/ts-2/wal
uuid: "391bb2553f8a44ee837bcd431341b7e5"
format_stamp: "Formatted at 2026-06-19 14:32:07 on dist-test-slave-70rm"
server_key: "c49d49f329526e4f348f62aeb25f367e"
server_key_iv: "69fa5c3936e1e67febf5fab9330ae868"
server_key_version: "encryptionkey@0"
I20260619 14:32:07.588678 4148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:07.605808 4148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:07.609154 4148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:32:07.609386 4148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:07.610952 4148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:32:07.611035 4148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:07.611104 4148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:32:07.611147 4148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 14:32:07 dist-test-slave-70rm krb5kdc[3751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879527, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.3@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:32:07.622267 4148 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.3:45137
I20260619 14:32:07.622285 4278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.3:45137 every 8 connection(s)
I20260619 14:32:07.623167 4148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
I20260619 14:32:07.624989 3845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:07.611901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.3:36931 (local address 127.2.2.62:42985)
0619 14:32:07.612059 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:07.612063 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:07.613144 (+ 1081us) server_negotiation.cc:407] Connection header received
0619 14:32:07.614234 (+ 1090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:07.614238 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:07.614299 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:07.614405 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:07.615631 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:07.616223 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:07.617387 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:07.617621 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:07.617738 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:07.621218 (+ 3480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:07.621228 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:07.621231 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:07.621256 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:07.622907 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:07.623520 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:07.623525 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:07.623527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:07.623637 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:07.624007 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:07.624010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:07.624011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:07.624199 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:07.624286 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:07.624769 (+ 483us) server_negotiation.cc:299] Negotiation successful
0619 14:32:07.624872 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 14:32:07.625855 4172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:07.612444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:42985 (local address 127.2.2.3:36931)
0619 14:32:07.612950 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:07.613026 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:07.613996 (+ 970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:07.614580 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:07.614587 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:07.614990 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:07.615479 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:07.615493 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:07.616347 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:07.616352 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:07.617252 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:07.617261 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:07.617733 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:07.617791 (+ 58us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:07.618016 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:07.618761 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:07.618777 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:07.621043 (+ 2266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:07.623052 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:07.623060 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:07.623076 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:07.623414 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:07.623743 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:07.623746 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:07.623748 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:07.623905 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:07.624318 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:07.624323 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:07.624556 (+ 233us) client_negotiation.cc:769] Sending connection context
0619 14:32:07.624717 (+ 161us) client_negotiation.cc:240] Negotiation successful
0619 14:32:07.624979 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":138,"threads_started":1}
I20260619 14:32:07.627203 4279 heartbeater.cc:347] Connected to a master server at 127.2.2.62:42985
I20260619 14:32:07.627368 4279 heartbeater.cc:464] Registering TS with master...
I20260619 14:32:07.627908 4279 heartbeater.cc:511] Master 127.2.2.62:42985 requested a full tablet report, sending...
I20260619 14:32:07.628947 3796 ts_manager.cc:194] Registered new tserver with Master: 391bb2553f8a44ee837bcd431341b7e5 (127.2.2.3:45137)
I20260619 14:32:07.629639 3796 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.2.3@KRBTEST.COM'} at 127.2.2.3:36931
I20260619 14:32:07.630332 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 4148
I20260619 14:32:07.630452 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/ts-2/wal/instance
I20260619 14:32:07.630779 2056 external_mini_cluster.cc:1428] Setting key eeb763d9037844651ea5488498751c54
I20260619 14:32:07.632958 2056 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:32:07.637648 3845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:07.629317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.3:42439 (local address 127.2.2.62:42985)
0619 14:32:07.629879 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:07.629882 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:07.629893 (+ 11us) server_negotiation.cc:407] Connection header received
0619 14:32:07.629920 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:07.629922 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:07.629951 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:07.630061 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:07.630626 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:07.631379 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:07.632425 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:07.632678 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:07.632781 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:07.634282 (+ 1501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:07.634288 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:07.634294 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:07.634325 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:07.635907 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:07.636428 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:07.636430 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:07.636432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:07.636504 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:07.636805 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:07.636807 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:07.636808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:07.636965 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:07.637103 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:07.637366 (+ 263us) server_negotiation.cc:299] Negotiation successful
0619 14:32:07.637472 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513}
I20260619 14:32:07.637661 4285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:07.629216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:42985 (local address 127.2.2.3:42439)
0619 14:32:07.629491 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:07.629503 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:07.629574 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:07.630061 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:07.630064 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:07.630304 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:07.630466 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:07.630474 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:07.631494 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:07.631497 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:07.632315 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:07.632323 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:07.632922 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:07.632939 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:07.633078 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:07.633622 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:07.633634 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:07.634091 (+ 457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:07.636038 (+ 1947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:07.636044 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:07.636049 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:07.636331 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:07.636594 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:07.636600 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:07.636606 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:07.636716 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:07.637094 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:07.637097 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:07.637200 (+ 103us) client_negotiation.cc:769] Sending connection context
0619 14:32:07.637304 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 14:32:07.637461 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":3,"thread_start_us":97,"threads_started":1}
Jun 19 14:32:07 dist-test-slave-70rm krb5kdc[3751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 14:32:07 dist-test-slave-70rm krb5kdc[3751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:32:07.659646 3845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:07.651320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59818 (local address 127.2.2.62:42985)
0619 14:32:07.651448 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:07.651451 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:07.651462 (+ 11us) server_negotiation.cc:407] Connection header received
0619 14:32:07.651505 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:07.651507 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:07.651545 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:07.651640 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:07.652335 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:07.652856 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:07.653706 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:07.653921 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:07.654079 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:07.656507 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:07.656510 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:07.656512 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:07.656544 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:07.657794 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:07.658282 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:07.658285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:07.658286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:07.658393 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:07.658670 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:07.658673 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:07.658674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:07.658876 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:07.659012 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:07.659274 (+ 262us) server_negotiation.cc:299] Negotiation successful
0619 14:32:07.659416 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 14:32:07.662685 3796 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59818:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:32:07.665678 3796 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:32:07.681628 4296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:07.674815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.2:38205 (local address 127.0.0.1:55986)
0619 14:32:07.675498 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:07.675550 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:07.675680 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:07.677230 (+ 1550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:07.677233 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:07.677246 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:32:07.677351 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:07.677359 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:07.678919 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:07.678923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:07.680190 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:07.680199 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:07.681203 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:07.681206 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:07.681293 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:07.681315 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 14:32:07.681380 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 14:32:07.681449 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"mutex_wait_us":7,"thread_start_us":93,"threads_started":1}
I20260619 14:32:07.681638 4297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:07.675019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55986 (local address 127.2.2.2:38205)
0619 14:32:07.676830 (+ 1811us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:07.676841 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:07.676863 (+ 22us) server_negotiation.cc:407] Connection header received
0619 14:32:07.676932 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:07.676937 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:07.677101 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:07.677239 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:32:07.677485 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:07.678643 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:07.680313 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:07.681098 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:07.681204 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:07.681353 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:07.681409 (+ 56us) server_negotiation.cc:299] Negotiation successful
0619 14:32:07.681497 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1714,"thread_start_us":104,"threads_started":1}
I20260619 14:32:07.681789 4301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:07.675647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52688 (local address 127.2.2.3:45137)
0619 14:32:07.676168 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:07.676176 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:07.676441 (+ 265us) server_negotiation.cc:407] Connection header received
0619 14:32:07.676647 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:07.676656 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:07.676911 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:07.677108 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:32:07.677421 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:07.678406 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:07.680048 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:07.680830 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:07.680943 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:07.681114 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:07.681538 (+ 424us) server_negotiation.cc:299] Negotiation successful
0619 14:32:07.681635 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":115,"threads_started":1}
I20260619 14:32:07.681986 4299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:07.675010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.3:45137 (local address 127.0.0.1:52688)
0619 14:32:07.676280 (+ 1270us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:07.676291 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:07.676478 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:07.677074 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:07.677079 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:07.677106 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:32:07.677294 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:07.677302 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:07.678509 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:07.678515 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:07.679936 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:07.679944 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:07.680961 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:07.680965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:07.681070 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:07.681121 (+ 51us) client_negotiation.cc:769] Sending connection context
0619 14:32:07.681239 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 14:32:07.681376 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1220,"mutex_wait_us":143,"thread_start_us":110,"threads_started":1}
I20260619 14:32:07.682339 4300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:07.675649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48050 (local address 127.2.2.1:42881)
0619 14:32:07.675900 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:07.675915 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:07.677591 (+ 1676us) server_negotiation.cc:407] Connection header received
0619 14:32:07.677676 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:07.677681 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:07.677868 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:07.678019 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:32:07.678245 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:07.679303 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:07.680591 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:07.681277 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:07.681367 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:07.681513 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:07.682053 (+ 540us) server_negotiation.cc:299] Negotiation successful
0619 14:32:07.682190 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":80,"threads_started":1}
I20260619 14:32:07.681968 4298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:07.675206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.1:42881 (local address 127.0.0.1:48050)
0619 14:32:07.677461 (+ 2255us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:07.677475 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:07.677581 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:07.678015 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:07.678017 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:07.678036 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:32:07.678150 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:07.678156 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:07.679441 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:07.679445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:07.680480 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:07.680486 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:07.681569 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:07.681572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:07.681667 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:07.681688 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 14:32:07.681757 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 14:32:07.681836 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2194,"thread_start_us":82,"threads_started":1}
I20260619 14:32:07.685256 3933 tablet_service.cc:1467] Processing CreateTablet for tablet 45983e43583d4fa99cc85852d3151213 (DEFAULT_TABLE table=test-table [id=916504edf5bd492399ab146cb415e02c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:32:07.685220 4213 tablet_service.cc:1467] Processing CreateTablet for tablet 45983e43583d4fa99cc85852d3151213 (DEFAULT_TABLE table=test-table [id=916504edf5bd492399ab146cb415e02c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:32:07.686393 3933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45983e43583d4fa99cc85852d3151213. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:07.686396 4213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45983e43583d4fa99cc85852d3151213. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:07.687790 4073 tablet_service.cc:1467] Processing CreateTablet for tablet 45983e43583d4fa99cc85852d3151213 (DEFAULT_TABLE table=test-table [id=916504edf5bd492399ab146cb415e02c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:32:07.688686 4073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45983e43583d4fa99cc85852d3151213. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:07.694044 4302 tablet_bootstrap.cc:492] T 45983e43583d4fa99cc85852d3151213 P 5bb5a7292e2248d28331461c809f2971: Bootstrap starting.
I20260619 14:32:07.694444 4303 tablet_bootstrap.cc:492] T 45983e43583d4fa99cc85852d3151213 P 391bb2553f8a44ee837bcd431341b7e5: Bootstrap starting.
I20260619 14:32:07.695268 4304 tablet_bootstrap.cc:492] T 45983e43583d4fa99cc85852d3151213 P ff059b5728f94491b9e2d7c8a97c641e: Bootstrap starting.
I20260619 14:32:07.697284 4304 tablet_bootstrap.cc:654] T 45983e43583d4fa99cc85852d3151213 P ff059b5728f94491b9e2d7c8a97c641e: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:07.697312 4303 tablet_bootstrap.cc:654] T 45983e43583d4fa99cc85852d3151213 P 391bb2553f8a44ee837bcd431341b7e5: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:07.698036 4304 log.cc:824] T 45983e43583d4fa99cc85852d3151213 P ff059b5728f94491b9e2d7c8a97c641e: Log is configured to *not* fsync() on all Append() calls
I20260619 14:32:07.698094 4303 log.cc:824] T 45983e43583d4fa99cc85852d3151213 P 391bb2553f8a44ee837bcd431341b7e5: Log is configured to *not* fsync() on all Append() calls
I20260619 14:32:07.698457 4302 tablet_bootstrap.cc:654] T 45983e43583d4fa99cc85852d3151213 P 5bb5a7292e2248d28331461c809f2971: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:07.699268 4302 log.cc:824] T 45983e43583d4fa99cc85852d3151213 P 5bb5a7292e2248d28331461c809f2971: Log is configured to *not* fsync() on all Append() calls
I20260619 14:32:07.700062 4303 tablet_bootstrap.cc:492] T 45983e43583d4fa99cc85852d3151213 P 391bb2553f8a44ee837bcd431341b7e5: No bootstrap required, opened a new log
I20260619 14:32:07.700142 4304 tablet_bootstrap.cc:492] T 45983e43583d4fa99cc85852d3151213 P ff059b5728f94491b9e2d7c8a97c641e: No bootstrap required, opened a new log
I20260619 14:32:07.700258 4303 ts_tablet_manager.cc:1397] T 45983e43583d4fa99cc85852d3151213 P 391bb2553f8a44ee837bcd431341b7e5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 14:32:07.700354 4304 ts_tablet_manager.cc:1397] T 45983e43583d4fa99cc85852d3151213 P ff059b5728f94491b9e2d7c8a97c641e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:32:07.701099 4302 tablet_bootstrap.cc:492] T 45983e43583d4fa99cc85852d3151213 P 5bb5a7292e2248d28331461c809f2971: No bootstrap required, opened a new log
I20260619 14:32:07.701279 4302 ts_tablet_manager.cc:1397] T 45983e43583d4fa99cc85852d3151213 P 5bb5a7292e2248d28331461c809f2971: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260619 14:32:07.703456 4304 raft_consensus.cc:348] T 45983e43583d4fa99cc85852d3151213 P ff059b5728f94491b9e2d7c8a97c641e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff059b5728f94491b9e2d7c8a97c641e" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38205 } } peers { permanent_uuid: "391bb2553f8a44ee837bcd431341b7e5" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45137 } } peers { permanent_uuid: "5bb5a7292e2248d28331461c809f2971" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 42881 } }
I20260619 14:32:07.703724 4304 raft_consensus.cc:374] T 45983e43583d4fa99cc85852d3151213 P ff059b5728f94491b9e2d7c8a97c641e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:07.703775 4304 raft_consensus.cc:729] T 45983e43583d4fa99cc85852d3151213 P ff059b5728f94491b9e2d7c8a97c641e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff059b5728f94491b9e2d7c8a97c641e, State: Initialized, Role: FOLLOWER
I20260619 14:32:07.704125 4303 raft_consensus.cc:348] T 45983e43583d4fa99cc85852d3151213 P 391bb2553f8a44ee837bcd431341b7e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff059b5728f94491b9e2d7c8a97c641e" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38205 } } peers { permanent_uuid: "391bb2553f8a44ee837bcd431341b7e5" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45137 } } peers { permanent_uuid: "5bb5a7292e2248d28331461c809f2971" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 42881 } }
I20260619 14:32:07.704229 4304 consensus_queue.cc:260] T 45983e43583d4fa99cc85852d3151213 P ff059b5728f94491b9e2d7c8a97c641e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff059b5728f94491b9e2d7c8a97c641e" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38205 } } peers { permanent_uuid: "391bb2553f8a44ee837bcd431341b7e5" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45137 } } peers { permanent_uuid: "5bb5a7292e2248d28331461c809f2971" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 42881 } }
I20260619 14:32:07.704367 4303 raft_consensus.cc:374] T 45983e43583d4fa99cc85852d3151213 P 391bb2553f8a44ee837bcd431341b7e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:07.704433 4303 raft_consensus.cc:729] T 45983e43583d4fa99cc85852d3151213 P 391bb2553f8a44ee837bcd431341b7e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 391bb2553f8a44ee837bcd431341b7e5, State: Initialized, Role: FOLLOWER
I20260619 14:32:07.704867 4303 consensus_queue.cc:260] T 45983e43583d4fa99cc85852d3151213 P 391bb2553f8a44ee837bcd431341b7e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff059b5728f94491b9e2d7c8a97c641e" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38205 } } peers { permanent_uuid: "391bb2553f8a44ee837bcd431341b7e5" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45137 } } peers { permanent_uuid: "5bb5a7292e2248d28331461c809f2971" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 42881 } }
I20260619 14:32:07.705224 4304 ts_tablet_manager.cc:1428] T 45983e43583d4fa99cc85852d3151213 P ff059b5728f94491b9e2d7c8a97c641e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 14:32:07.705564 4139 heartbeater.cc:503] Master 127.2.2.62:42985 was elected leader, sending a full tablet report...
I20260619 14:32:07.705564 4279 heartbeater.cc:503] Master 127.2.2.62:42985 was elected leader, sending a full tablet report...
I20260619 14:32:07.705444 4302 raft_consensus.cc:348] T 45983e43583d4fa99cc85852d3151213 P 5bb5a7292e2248d28331461c809f2971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff059b5728f94491b9e2d7c8a97c641e" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38205 } } peers { permanent_uuid: "391bb2553f8a44ee837bcd431341b7e5" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45137 } } peers { permanent_uuid: "5bb5a7292e2248d28331461c809f2971" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 42881 } }
I20260619 14:32:07.705668 4302 raft_consensus.cc:374] T 45983e43583d4fa99cc85852d3151213 P 5bb5a7292e2248d28331461c809f2971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:07.705724 4302 raft_consensus.cc:729] T 45983e43583d4fa99cc85852d3151213 P 5bb5a7292e2248d28331461c809f2971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bb5a7292e2248d28331461c809f2971, State: Initialized, Role: FOLLOWER
I20260619 14:32:07.705829 4303 ts_tablet_manager.cc:1428] T 45983e43583d4fa99cc85852d3151213 P 391bb2553f8a44ee837bcd431341b7e5: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 14:32:07.706159 4302 consensus_queue.cc:260] T 45983e43583d4fa99cc85852d3151213 P 5bb5a7292e2248d28331461c809f2971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff059b5728f94491b9e2d7c8a97c641e" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38205 } } peers { permanent_uuid: "391bb2553f8a44ee837bcd431341b7e5" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45137 } } peers { permanent_uuid: "5bb5a7292e2248d28331461c809f2971" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 42881 } }
I20260619 14:32:07.706816 3999 heartbeater.cc:503] Master 127.2.2.62:42985 was elected leader, sending a full tablet report...
I20260619 14:32:07.707196 4302 ts_tablet_manager.cc:1428] T 45983e43583d4fa99cc85852d3151213 P 5bb5a7292e2248d28331461c809f2971: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 14:32:07.777715 4308 raft_consensus.cc:482] T 45983e43583d4fa99cc85852d3151213 P ff059b5728f94491b9e2d7c8a97c641e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:32:07.777937 4308 raft_consensus.cc:504] T 45983e43583d4fa99cc85852d3151213 P ff059b5728f94491b9e2d7c8a97c641e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff059b5728f94491b9e2d7c8a97c641e" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38205 } } peers { permanent_uuid: "391bb2553f8a44ee837bcd431341b7e5" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45137 } } peers { permanent_uuid: "5bb5a7292e2248d28331461c809f2971" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 42881 } }
I20260619 14:32:07.779193 4308 leader_election.cc:288] T 45983e43583d4fa99cc85852d3151213 P ff059b5728f94491b9e2d7c8a97c641e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 391bb2553f8a44ee837bcd431341b7e5 (127.2.2.3:45137), 5bb5a7292e2248d28331461c809f2971 (127.2.2.1:42881)
I20260619 14:32:07.783403 4301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:07.779479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:56001 (local address 127.2.2.3:45137)
0619 14:32:07.779622 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:07.779627 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:07.779734 (+ 107us) server_negotiation.cc:407] Connection header received
0619 14:32:07.779889 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:07.779893 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:07.779956 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:07.780048 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:32:07.780370 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:07.780944 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:07.782098 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:07.782745 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:07.782874 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:07.783013 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:07.783230 (+ 217us) server_negotiation.cc:299] Negotiation successful
0619 14:32:07.783294 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 14:32:07.783409 4032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:07.779397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.3:45137 (local address 127.2.2.2:56001)
0619 14:32:07.779632 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:07.779650 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:07.779761 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:07.780066 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:07.780071 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:07.780096 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:32:07.780253 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:07.780260 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:07.781090 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:07.781093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:07.781949 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:07.781958 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:07.782887 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:07.782890 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:07.783079 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:07.783106 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 14:32:07.783193 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 14:32:07.783273 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144}
I20260619 14:32:07.784134 4300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:07.779591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:50483 (local address 127.2.2.1:42881)
0619 14:32:07.779738 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:07.779742 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:07.780040 (+ 298us) server_negotiation.cc:407] Connection header received
0619 14:32:07.780121 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:07.780124 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:07.780172 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:07.780253 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:32:07.780594 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:07.781173 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:07.782991 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:07.783601 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:07.783712 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:07.783770 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:07.783951 (+ 181us) server_negotiation.cc:299] Negotiation successful
0619 14:32:07.784024 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 14:32:07.784147 4311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:07.779479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.1:42881 (local address 127.2.2.2:50483)
0619 14:32:07.779914 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:07.779926 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:07.780033 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:07.780274 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:07.780276 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:07.780289 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:32:07.780434 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:07.780440 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:07.781313 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:07.781316 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:07.782851 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:07.782861 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:07.783718 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:07.783721 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:07.783815 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:07.783835 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 14:32:07.783935 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 14:32:07.784008 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":150,"threads_started":1}
I20260619 14:32:07.784221 4233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45983e43583d4fa99cc85852d3151213" candidate_uuid: "ff059b5728f94491b9e2d7c8a97c641e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "391bb2553f8a44ee837bcd431341b7e5" is_pre_election: true
I20260619 14:32:07.784600 4233 raft_consensus.cc:2454] T 45983e43583d4fa99cc85852d3151213 P 391bb2553f8a44ee837bcd431341b7e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff059b5728f94491b9e2d7c8a97c641e in term 0.
I20260619 14:32:07.784888 3953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45983e43583d4fa99cc85852d3151213" candidate_uuid: "ff059b5728f94491b9e2d7c8a97c641e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bb5a7292e2248d28331461c809f2971" is_pre_election: true
I20260619 14:32:07.785200 3953 raft_consensus.cc:2454] T 45983e43583d4fa99cc85852d3151213 P 5bb5a7292e2248d28331461c809f2971 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff059b5728f94491b9e2d7c8a97c641e in term 0.
I20260619 14:32:07.785168 4027 leader_election.cc:302] T 45983e43583d4fa99cc85852d3151213 P ff059b5728f94491b9e2d7c8a97c641e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 391bb2553f8a44ee837bcd431341b7e5, ff059b5728f94491b9e2d7c8a97c641e; no voters:
I20260619 14:32:07.785499 4308 raft_consensus.cc:2781] T 45983e43583d4fa99cc85852d3151213 P ff059b5728f94491b9e2d7c8a97c641e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:32:07.785610 4308 raft_consensus.cc:482] T 45983e43583d4fa99cc85852d3151213 P ff059b5728f94491b9e2d7c8a97c641e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:32:07.785678 4308 raft_consensus.cc:3037] T 45983e43583d4fa99cc85852d3151213 P ff059b5728f94491b9e2d7c8a97c641e [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:07.786895 4308 raft_consensus.cc:504] T 45983e43583d4fa99cc85852d3151213 P ff059b5728f94491b9e2d7c8a97c641e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff059b5728f94491b9e2d7c8a97c641e" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38205 } } peers { permanent_uuid: "391bb2553f8a44ee837bcd431341b7e5" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45137 } } peers { permanent_uuid: "5bb5a7292e2248d28331461c809f2971" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 42881 } }
I20260619 14:32:07.787323 4308 leader_election.cc:288] T 45983e43583d4fa99cc85852d3151213 P ff059b5728f94491b9e2d7c8a97c641e [CANDIDATE]: Term 1 election: Requested vote from peers 391bb2553f8a44ee837bcd431341b7e5 (127.2.2.3:45137), 5bb5a7292e2248d28331461c809f2971 (127.2.2.1:42881)
I20260619 14:32:07.787707 4233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45983e43583d4fa99cc85852d3151213" candidate_uuid: "ff059b5728f94491b9e2d7c8a97c641e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "391bb2553f8a44ee837bcd431341b7e5"
I20260619 14:32:07.787715 3953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45983e43583d4fa99cc85852d3151213" candidate_uuid: "ff059b5728f94491b9e2d7c8a97c641e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bb5a7292e2248d28331461c809f2971"
I20260619 14:32:07.787845 4233 raft_consensus.cc:3037] T 45983e43583d4fa99cc85852d3151213 P 391bb2553f8a44ee837bcd431341b7e5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:07.787846 3953 raft_consensus.cc:3037] T 45983e43583d4fa99cc85852d3151213 P 5bb5a7292e2248d28331461c809f2971 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:07.789217 3953 raft_consensus.cc:2454] T 45983e43583d4fa99cc85852d3151213 P 5bb5a7292e2248d28331461c809f2971 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff059b5728f94491b9e2d7c8a97c641e in term 1.
I20260619 14:32:07.789217 4233 raft_consensus.cc:2454] T 45983e43583d4fa99cc85852d3151213 P 391bb2553f8a44ee837bcd431341b7e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff059b5728f94491b9e2d7c8a97c641e in term 1.
I20260619 14:32:07.789553 4027 leader_election.cc:302] T 45983e43583d4fa99cc85852d3151213 P ff059b5728f94491b9e2d7c8a97c641e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 391bb2553f8a44ee837bcd431341b7e5, 5bb5a7292e2248d28331461c809f2971, ff059b5728f94491b9e2d7c8a97c641e; no voters:
I20260619 14:32:07.789741 4308 raft_consensus.cc:2781] T 45983e43583d4fa99cc85852d3151213 P ff059b5728f94491b9e2d7c8a97c641e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:32:07.789968 4308 raft_consensus.cc:686] T 45983e43583d4fa99cc85852d3151213 P ff059b5728f94491b9e2d7c8a97c641e [term 1 LEADER]: Becoming Leader. State: Replica: ff059b5728f94491b9e2d7c8a97c641e, State: Running, Role: LEADER
I20260619 14:32:07.790340 4308 consensus_queue.cc:237] T 45983e43583d4fa99cc85852d3151213 P ff059b5728f94491b9e2d7c8a97c641e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff059b5728f94491b9e2d7c8a97c641e" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38205 } } peers { permanent_uuid: "391bb2553f8a44ee837bcd431341b7e5" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45137 } } peers { permanent_uuid: "5bb5a7292e2248d28331461c809f2971" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 42881 } }
I20260619 14:32:07.793735 3794 catalog_manager.cc:5495] T 45983e43583d4fa99cc85852d3151213 P ff059b5728f94491b9e2d7c8a97c641e reported cstate change: term changed from 0 to 1, leader changed from <none> to ff059b5728f94491b9e2d7c8a97c641e (127.2.2.2). New cstate: current_term: 1 leader_uuid: "ff059b5728f94491b9e2d7c8a97c641e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff059b5728f94491b9e2d7c8a97c641e" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "391bb2553f8a44ee837bcd431341b7e5" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5bb5a7292e2248d28331461c809f2971" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 42881 } health_report { overall_health: UNKNOWN } } }
Jun 19 14:32:07 dist-test-slave-70rm krb5kdc[3751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879526, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.62@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:32:07.856413 4319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:07.847887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:42985 (local address 127.0.0.1:59826)
0619 14:32:07.848202 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:07.848219 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:07.848332 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:07.848583 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:07.848586 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:07.848979 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:07.849070 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:07.849075 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:07.849811 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:07.849814 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:07.850509 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:07.850518 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:07.850991 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:07.851009 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:07.851069 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:07.851612 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:07.851624 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:07.853068 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:07.854829 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:07.854838 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:07.854844 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:07.855076 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:07.855311 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:07.855313 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:07.855314 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:07.855389 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:07.855865 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:07.855871 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:07.855981 (+ 110us) client_negotiation.cc:769] Sending connection context
0619 14:32:07.856090 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 14:32:07.856220 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":149,"threads_started":1}
I20260619 14:32:07.856412 3845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:07.847931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59826 (local address 127.2.2.62:42985)
0619 14:32:07.848104 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:07.848108 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:07.848295 (+ 187us) server_negotiation.cc:407] Connection header received
0619 14:32:07.848411 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:07.848413 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:07.848453 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:07.848548 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:07.849171 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:07.849719 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:07.850615 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:07.850889 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:07.850975 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:07.853214 (+ 2239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:07.853217 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:07.853219 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:07.853239 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:07.854726 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:07.855161 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:07.855163 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:07.855165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:07.855236 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:07.855462 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:07.855465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:07.855466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:07.855643 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:07.855718 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:07.856147 (+ 429us) server_negotiation.cc:299] Negotiation successful
0619 14:32:07.856277 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":91}
I20260619 14:32:07.860762 3794 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:59826:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 14:32:07.861510 3794 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:32:07.866703 4073 tablet_service.cc:1467] Processing CreateTablet for tablet ac9bb167dc6e42d98725c8eacec0cd51 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a9ac190b197c485da66ad476d20540b6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 14:32:07.866667 3933 tablet_service.cc:1467] Processing CreateTablet for tablet ac9bb167dc6e42d98725c8eacec0cd51 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a9ac190b197c485da66ad476d20540b6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 14:32:07.867413 4073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac9bb167dc6e42d98725c8eacec0cd51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:07.867424 3933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac9bb167dc6e42d98725c8eacec0cd51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:07.867193 4213 tablet_service.cc:1467] Processing CreateTablet for tablet ac9bb167dc6e42d98725c8eacec0cd51 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a9ac190b197c485da66ad476d20540b6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 14:32:07.867839 4213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac9bb167dc6e42d98725c8eacec0cd51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:07.870189 4304 tablet_bootstrap.cc:492] T ac9bb167dc6e42d98725c8eacec0cd51 P ff059b5728f94491b9e2d7c8a97c641e: Bootstrap starting.
I20260619 14:32:07.870756 4302 tablet_bootstrap.cc:492] T ac9bb167dc6e42d98725c8eacec0cd51 P 5bb5a7292e2248d28331461c809f2971: Bootstrap starting.
I20260619 14:32:07.871266 4304 tablet_bootstrap.cc:654] T ac9bb167dc6e42d98725c8eacec0cd51 P ff059b5728f94491b9e2d7c8a97c641e: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:07.871723 4302 tablet_bootstrap.cc:654] T ac9bb167dc6e42d98725c8eacec0cd51 P 5bb5a7292e2248d28331461c809f2971: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:07.871946 4303 tablet_bootstrap.cc:492] T ac9bb167dc6e42d98725c8eacec0cd51 P 391bb2553f8a44ee837bcd431341b7e5: Bootstrap starting.
I20260619 14:32:07.872571 4304 tablet_bootstrap.cc:492] T ac9bb167dc6e42d98725c8eacec0cd51 P ff059b5728f94491b9e2d7c8a97c641e: No bootstrap required, opened a new log
I20260619 14:32:07.872640 4304 ts_tablet_manager.cc:1397] T ac9bb167dc6e42d98725c8eacec0cd51 P ff059b5728f94491b9e2d7c8a97c641e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 14:32:07.872768 4303 tablet_bootstrap.cc:654] T ac9bb167dc6e42d98725c8eacec0cd51 P 391bb2553f8a44ee837bcd431341b7e5: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:07.873049 4304 raft_consensus.cc:348] T ac9bb167dc6e42d98725c8eacec0cd51 P ff059b5728f94491b9e2d7c8a97c641e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb5a7292e2248d28331461c809f2971" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 42881 } } peers { permanent_uuid: "ff059b5728f94491b9e2d7c8a97c641e" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38205 } } peers { permanent_uuid: "391bb2553f8a44ee837bcd431341b7e5" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45137 } }
I20260619 14:32:07.873140 4304 raft_consensus.cc:374] T ac9bb167dc6e42d98725c8eacec0cd51 P ff059b5728f94491b9e2d7c8a97c641e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:07.873164 4304 raft_consensus.cc:729] T ac9bb167dc6e42d98725c8eacec0cd51 P ff059b5728f94491b9e2d7c8a97c641e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff059b5728f94491b9e2d7c8a97c641e, State: Initialized, Role: FOLLOWER
I20260619 14:32:07.873347 4304 consensus_queue.cc:260] T ac9bb167dc6e42d98725c8eacec0cd51 P ff059b5728f94491b9e2d7c8a97c641e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb5a7292e2248d28331461c809f2971" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 42881 } } peers { permanent_uuid: "ff059b5728f94491b9e2d7c8a97c641e" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38205 } } peers { permanent_uuid: "391bb2553f8a44ee837bcd431341b7e5" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45137 } }
I20260619 14:32:07.873620 4308 tablet_replica.cc:439] T ac9bb167dc6e42d98725c8eacec0cd51 P ff059b5728f94491b9e2d7c8a97c641e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb5a7292e2248d28331461c809f2971" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 42881 } } peers { permanent_uuid: "ff059b5728f94491b9e2d7c8a97c641e" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38205 } } peers { permanent_uuid: "391bb2553f8a44ee837bcd431341b7e5" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45137 } } }
I20260619 14:32:07.873678 4304 ts_tablet_manager.cc:1428] T ac9bb167dc6e42d98725c8eacec0cd51 P ff059b5728f94491b9e2d7c8a97c641e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 14:32:07.873768 4308 tablet_replica.cc:442] T ac9bb167dc6e42d98725c8eacec0cd51 P ff059b5728f94491b9e2d7c8a97c641e: This TxnStatusTablet replica's current role is: FOLLOWER
W20260619 14:32:07.874181 4280 tablet.cc:2369] T 45983e43583d4fa99cc85852d3151213 P 391bb2553f8a44ee837bcd431341b7e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:32:07.874873 4302 tablet_bootstrap.cc:492] T ac9bb167dc6e42d98725c8eacec0cd51 P 5bb5a7292e2248d28331461c809f2971: No bootstrap required, opened a new log
I20260619 14:32:07.874899 4303 tablet_bootstrap.cc:492] T ac9bb167dc6e42d98725c8eacec0cd51 P 391bb2553f8a44ee837bcd431341b7e5: No bootstrap required, opened a new log
I20260619 14:32:07.874989 4303 ts_tablet_manager.cc:1397] T ac9bb167dc6e42d98725c8eacec0cd51 P 391bb2553f8a44ee837bcd431341b7e5: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 14:32:07.874994 4302 ts_tablet_manager.cc:1397] T ac9bb167dc6e42d98725c8eacec0cd51 P 5bb5a7292e2248d28331461c809f2971: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260619 14:32:07.875456 4302 raft_consensus.cc:348] T ac9bb167dc6e42d98725c8eacec0cd51 P 5bb5a7292e2248d28331461c809f2971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb5a7292e2248d28331461c809f2971" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 42881 } } peers { permanent_uuid: "ff059b5728f94491b9e2d7c8a97c641e" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38205 } } peers { permanent_uuid: "391bb2553f8a44ee837bcd431341b7e5" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45137 } }
I20260619 14:32:07.875492 4303 raft_consensus.cc:348] T ac9bb167dc6e42d98725c8eacec0cd51 P 391bb2553f8a44ee837bcd431341b7e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb5a7292e2248d28331461c809f2971" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 42881 } } peers { permanent_uuid: "ff059b5728f94491b9e2d7c8a97c641e" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38205 } } peers { permanent_uuid: "391bb2553f8a44ee837bcd431341b7e5" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45137 } }
I20260619 14:32:07.875584 4302 raft_consensus.cc:374] T ac9bb167dc6e42d98725c8eacec0cd51 P 5bb5a7292e2248d28331461c809f2971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:07.875600 4303 raft_consensus.cc:374] T ac9bb167dc6e42d98725c8eacec0cd51 P 391bb2553f8a44ee837bcd431341b7e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:07.875636 4303 raft_consensus.cc:729] T ac9bb167dc6e42d98725c8eacec0cd51 P 391bb2553f8a44ee837bcd431341b7e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 391bb2553f8a44ee837bcd431341b7e5, State: Initialized, Role: FOLLOWER
I20260619 14:32:07.875630 4302 raft_consensus.cc:729] T ac9bb167dc6e42d98725c8eacec0cd51 P 5bb5a7292e2248d28331461c809f2971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bb5a7292e2248d28331461c809f2971, State: Initialized, Role: FOLLOWER
I20260619 14:32:07.875758 4303 consensus_queue.cc:260] T ac9bb167dc6e42d98725c8eacec0cd51 P 391bb2553f8a44ee837bcd431341b7e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb5a7292e2248d28331461c809f2971" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 42881 } } peers { permanent_uuid: "ff059b5728f94491b9e2d7c8a97c641e" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38205 } } peers { permanent_uuid: "391bb2553f8a44ee837bcd431341b7e5" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45137 } }
I20260619 14:32:07.875772 4302 consensus_queue.cc:260] T ac9bb167dc6e42d98725c8eacec0cd51 P 5bb5a7292e2248d28331461c809f2971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb5a7292e2248d28331461c809f2971" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 42881 } } peers { permanent_uuid: "ff059b5728f94491b9e2d7c8a97c641e" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38205 } } peers { permanent_uuid: "391bb2553f8a44ee837bcd431341b7e5" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45137 } }
I20260619 14:32:07.876061 4302 ts_tablet_manager.cc:1428] T ac9bb167dc6e42d98725c8eacec0cd51 P 5bb5a7292e2248d28331461c809f2971: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:32:07.876027 4309 tablet_replica.cc:439] T ac9bb167dc6e42d98725c8eacec0cd51 P 391bb2553f8a44ee837bcd431341b7e5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb5a7292e2248d28331461c809f2971" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 42881 } } peers { permanent_uuid: "ff059b5728f94491b9e2d7c8a97c641e" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38205 } } peers { permanent_uuid: "391bb2553f8a44ee837bcd431341b7e5" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45137 } } }
I20260619 14:32:07.876153 4309 tablet_replica.cc:442] T ac9bb167dc6e42d98725c8eacec0cd51 P 391bb2553f8a44ee837bcd431341b7e5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:32:07.876356 4310 tablet_replica.cc:439] T ac9bb167dc6e42d98725c8eacec0cd51 P 5bb5a7292e2248d28331461c809f2971: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb5a7292e2248d28331461c809f2971" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 42881 } } peers { permanent_uuid: "ff059b5728f94491b9e2d7c8a97c641e" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38205 } } peers { permanent_uuid: "391bb2553f8a44ee837bcd431341b7e5" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45137 } } }
I20260619 14:32:07.876492 4310 tablet_replica.cc:442] T ac9bb167dc6e42d98725c8eacec0cd51 P 5bb5a7292e2248d28331461c809f2971: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:32:07.876039 4303 ts_tablet_manager.cc:1428] T ac9bb167dc6e42d98725c8eacec0cd51 P 391bb2553f8a44ee837bcd431341b7e5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20260619 14:32:07.899825 4140 tablet.cc:2369] T 45983e43583d4fa99cc85852d3151213 P ff059b5728f94491b9e2d7c8a97c641e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:32:07.933578 4000 tablet.cc:2369] T 45983e43583d4fa99cc85852d3151213 P 5bb5a7292e2248d28331461c809f2971: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:32:08.072487 4309 raft_consensus.cc:482] T ac9bb167dc6e42d98725c8eacec0cd51 P 391bb2553f8a44ee837bcd431341b7e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:32:08.072636 4309 raft_consensus.cc:504] T ac9bb167dc6e42d98725c8eacec0cd51 P 391bb2553f8a44ee837bcd431341b7e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb5a7292e2248d28331461c809f2971" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 42881 } } peers { permanent_uuid: "ff059b5728f94491b9e2d7c8a97c641e" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38205 } } peers { permanent_uuid: "391bb2553f8a44ee837bcd431341b7e5" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45137 } }
I20260619 14:32:08.073717 4309 leader_election.cc:288] T ac9bb167dc6e42d98725c8eacec0cd51 P 391bb2553f8a44ee837bcd431341b7e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5bb5a7292e2248d28331461c809f2971 (127.2.2.1:42881), ff059b5728f94491b9e2d7c8a97c641e (127.2.2.2:38205)
I20260619 14:32:08.077435 4172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:08.073997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.2:38205 (local address 127.2.2.3:39111)
0619 14:32:08.074125 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:08.074167 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:08.074270 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:08.074587 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:08.074590 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:08.074615 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:32:08.074736 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:08.074743 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:08.075527 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:08.075532 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:08.076324 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:08.076333 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:08.076980 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:08.076984 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:08.077081 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:08.077114 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 14:32:08.077201 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 14:32:08.077278 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260619 14:32:08.077417 4297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:08.074055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.3:39111 (local address 127.2.2.2:38205)
0619 14:32:08.074216 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:08.074221 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:08.074237 (+ 16us) server_negotiation.cc:407] Connection header received
0619 14:32:08.074396 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:08.074399 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:08.074454 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:08.074536 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:32:08.074895 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:08.075406 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:08.076451 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:08.076854 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:08.076978 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:08.077041 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:08.077215 (+ 174us) server_negotiation.cc:299] Negotiation successful
0619 14:32:08.077282 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 14:32:08.077920 4300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:08.074166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.3:50041 (local address 127.2.2.1:42881)
0619 14:32:08.074313 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:08.074316 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:08.074841 (+ 525us) server_negotiation.cc:407] Connection header received
0619 14:32:08.074981 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:08.074984 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:08.075024 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:08.075108 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:32:08.075378 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:08.075864 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:08.076903 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:08.077299 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:08.077424 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:08.077482 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:08.077695 (+ 213us) server_negotiation.cc:299] Negotiation successful
0619 14:32:08.077773 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 14:32:08.078097 4093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9bb167dc6e42d98725c8eacec0cd51" candidate_uuid: "391bb2553f8a44ee837bcd431341b7e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff059b5728f94491b9e2d7c8a97c641e" is_pre_election: true
I20260619 14:32:08.078341 4323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:08.074001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.1:42881 (local address 127.2.2.3:50041)
0619 14:32:08.074748 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:08.074761 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:08.074858 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:08.075118 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:08.075121 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:08.075136 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:32:08.075265 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:08.075270 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:08.075972 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:08.075979 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:08.076763 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:08.076772 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:08.077432 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:08.077436 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:08.077544 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:08.077570 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 14:32:08.078085 (+ 515us) client_negotiation.cc:240] Negotiation successful
0619 14:32:08.078195 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"mutex_wait_us":374,"thread_start_us":140,"threads_started":1}
I20260619 14:32:08.078400 4093 raft_consensus.cc:2454] T ac9bb167dc6e42d98725c8eacec0cd51 P ff059b5728f94491b9e2d7c8a97c641e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 391bb2553f8a44ee837bcd431341b7e5 in term 0.
I20260619 14:32:08.078795 3953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9bb167dc6e42d98725c8eacec0cd51" candidate_uuid: "391bb2553f8a44ee837bcd431341b7e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bb5a7292e2248d28331461c809f2971" is_pre_election: true
I20260619 14:32:08.078930 3953 raft_consensus.cc:2454] T ac9bb167dc6e42d98725c8eacec0cd51 P 5bb5a7292e2248d28331461c809f2971 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 391bb2553f8a44ee837bcd431341b7e5 in term 0.
I20260619 14:32:08.078914 4166 leader_election.cc:302] T ac9bb167dc6e42d98725c8eacec0cd51 P 391bb2553f8a44ee837bcd431341b7e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 391bb2553f8a44ee837bcd431341b7e5, ff059b5728f94491b9e2d7c8a97c641e; no voters:
I20260619 14:32:08.079229 4309 raft_consensus.cc:2781] T ac9bb167dc6e42d98725c8eacec0cd51 P 391bb2553f8a44ee837bcd431341b7e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:32:08.079296 4309 raft_consensus.cc:482] T ac9bb167dc6e42d98725c8eacec0cd51 P 391bb2553f8a44ee837bcd431341b7e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:32:08.079334 4309 raft_consensus.cc:3037] T ac9bb167dc6e42d98725c8eacec0cd51 P 391bb2553f8a44ee837bcd431341b7e5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:08.080338 4309 raft_consensus.cc:504] T ac9bb167dc6e42d98725c8eacec0cd51 P 391bb2553f8a44ee837bcd431341b7e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb5a7292e2248d28331461c809f2971" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 42881 } } peers { permanent_uuid: "ff059b5728f94491b9e2d7c8a97c641e" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38205 } } peers { permanent_uuid: "391bb2553f8a44ee837bcd431341b7e5" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45137 } }
I20260619 14:32:08.080688 4309 leader_election.cc:288] T ac9bb167dc6e42d98725c8eacec0cd51 P 391bb2553f8a44ee837bcd431341b7e5 [CANDIDATE]: Term 1 election: Requested vote from peers 5bb5a7292e2248d28331461c809f2971 (127.2.2.1:42881), ff059b5728f94491b9e2d7c8a97c641e (127.2.2.2:38205)
I20260619 14:32:08.081013 3953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9bb167dc6e42d98725c8eacec0cd51" candidate_uuid: "391bb2553f8a44ee837bcd431341b7e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bb5a7292e2248d28331461c809f2971"
I20260619 14:32:08.081077 4093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9bb167dc6e42d98725c8eacec0cd51" candidate_uuid: "391bb2553f8a44ee837bcd431341b7e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff059b5728f94491b9e2d7c8a97c641e"
I20260619 14:32:08.081142 3953 raft_consensus.cc:3037] T ac9bb167dc6e42d98725c8eacec0cd51 P 5bb5a7292e2248d28331461c809f2971 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:08.081225 4093 raft_consensus.cc:3037] T ac9bb167dc6e42d98725c8eacec0cd51 P ff059b5728f94491b9e2d7c8a97c641e [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:08.082087 3953 raft_consensus.cc:2454] T ac9bb167dc6e42d98725c8eacec0cd51 P 5bb5a7292e2248d28331461c809f2971 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 391bb2553f8a44ee837bcd431341b7e5 in term 1.
I20260619 14:32:08.082244 4093 raft_consensus.cc:2454] T ac9bb167dc6e42d98725c8eacec0cd51 P ff059b5728f94491b9e2d7c8a97c641e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 391bb2553f8a44ee837bcd431341b7e5 in term 1.
I20260619 14:32:08.082458 4165 leader_election.cc:302] T ac9bb167dc6e42d98725c8eacec0cd51 P 391bb2553f8a44ee837bcd431341b7e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 391bb2553f8a44ee837bcd431341b7e5, 5bb5a7292e2248d28331461c809f2971; no voters:
I20260619 14:32:08.082655 4309 raft_consensus.cc:2781] T ac9bb167dc6e42d98725c8eacec0cd51 P 391bb2553f8a44ee837bcd431341b7e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:32:08.082857 4309 raft_consensus.cc:686] T ac9bb167dc6e42d98725c8eacec0cd51 P 391bb2553f8a44ee837bcd431341b7e5 [term 1 LEADER]: Becoming Leader. State: Replica: 391bb2553f8a44ee837bcd431341b7e5, State: Running, Role: LEADER
I20260619 14:32:08.083238 4309 consensus_queue.cc:237] T ac9bb167dc6e42d98725c8eacec0cd51 P 391bb2553f8a44ee837bcd431341b7e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb5a7292e2248d28331461c809f2971" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 42881 } } peers { permanent_uuid: "ff059b5728f94491b9e2d7c8a97c641e" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38205 } } peers { permanent_uuid: "391bb2553f8a44ee837bcd431341b7e5" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45137 } }
I20260619 14:32:08.085543 4324 tablet_replica.cc:439] T ac9bb167dc6e42d98725c8eacec0cd51 P 391bb2553f8a44ee837bcd431341b7e5: TxnStatusTablet state changed. Reason: New leader 391bb2553f8a44ee837bcd431341b7e5. Latest consensus state: current_term: 1 leader_uuid: "391bb2553f8a44ee837bcd431341b7e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb5a7292e2248d28331461c809f2971" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 42881 } } peers { permanent_uuid: "ff059b5728f94491b9e2d7c8a97c641e" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38205 } } peers { permanent_uuid: "391bb2553f8a44ee837bcd431341b7e5" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45137 } } }
I20260619 14:32:08.085656 4324 tablet_replica.cc:442] T ac9bb167dc6e42d98725c8eacec0cd51 P 391bb2553f8a44ee837bcd431341b7e5: This TxnStatusTablet replica's current role is: LEADER
I20260619 14:32:08.086067 4326 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 14:32:08.086303 4326 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 14:32:08.086629 3795 catalog_manager.cc:5495] T ac9bb167dc6e42d98725c8eacec0cd51 P 391bb2553f8a44ee837bcd431341b7e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 391bb2553f8a44ee837bcd431341b7e5 (127.2.2.3). New cstate: current_term: 1 leader_uuid: "391bb2553f8a44ee837bcd431341b7e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb5a7292e2248d28331461c809f2971" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 42881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff059b5728f94491b9e2d7c8a97c641e" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "391bb2553f8a44ee837bcd431341b7e5" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45137 } health_report { overall_health: HEALTHY } } }
I20260619 14:32:08.116904 4301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:08.112923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52702 (local address 127.2.2.3:45137)
0619 14:32:08.113056 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:08.113060 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:08.113072 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:32:08.113149 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:08.113151 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:08.113203 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:08.113287 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:32:08.113569 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:08.114060 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:08.115371 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:08.115620 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:08.115701 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:08.116037 (+ 336us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:32:08.116490 (+ 453us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:32:08.116575 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:08.116730 (+ 155us) server_negotiation.cc:299] Negotiation successful
0619 14:32:08.116791 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 14:32:08.116938 4319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:08.112698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.3:45137 (local address 127.0.0.1:52702)
0619 14:32:08.112857 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:08.112874 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:08.112993 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:08.113329 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:08.113332 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:08.113340 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:32:08.113452 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:08.113458 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:08.114243 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:08.114248 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:08.115231 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:08.115242 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:08.115712 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:08.115731 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:08.115832 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:08.115894 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:32:08.116585 (+ 691us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:32:08.116620 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 14:32:08.116696 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 14:32:08.116790 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 14:32:08.122751 3953 raft_consensus.cc:1261] T ac9bb167dc6e42d98725c8eacec0cd51 P 5bb5a7292e2248d28331461c809f2971 [term 1 FOLLOWER]: Refusing update from remote peer 391bb2553f8a44ee837bcd431341b7e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:32:08.122751 4093 raft_consensus.cc:1261] T ac9bb167dc6e42d98725c8eacec0cd51 P ff059b5728f94491b9e2d7c8a97c641e [term 1 FOLLOWER]: Refusing update from remote peer 391bb2553f8a44ee837bcd431341b7e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:32:08.123467 4309 consensus_queue.cc:1035] T ac9bb167dc6e42d98725c8eacec0cd51 P 391bb2553f8a44ee837bcd431341b7e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff059b5728f94491b9e2d7c8a97c641e" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:32:08.123656 4324 consensus_queue.cc:1035] T ac9bb167dc6e42d98725c8eacec0cd51 P 391bb2553f8a44ee837bcd431341b7e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bb5a7292e2248d28331461c809f2971" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 42881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:32:08.127128 4308 tablet_replica.cc:439] T ac9bb167dc6e42d98725c8eacec0cd51 P ff059b5728f94491b9e2d7c8a97c641e: TxnStatusTablet state changed. Reason: New leader 391bb2553f8a44ee837bcd431341b7e5. Latest consensus state: current_term: 1 leader_uuid: "391bb2553f8a44ee837bcd431341b7e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb5a7292e2248d28331461c809f2971" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 42881 } } peers { permanent_uuid: "ff059b5728f94491b9e2d7c8a97c641e" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38205 } } peers { permanent_uuid: "391bb2553f8a44ee837bcd431341b7e5" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45137 } } }
I20260619 14:32:08.127254 4308 tablet_replica.cc:442] T ac9bb167dc6e42d98725c8eacec0cd51 P ff059b5728f94491b9e2d7c8a97c641e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:32:08.128295 4310 tablet_replica.cc:439] T ac9bb167dc6e42d98725c8eacec0cd51 P 5bb5a7292e2248d28331461c809f2971: TxnStatusTablet state changed. Reason: New leader 391bb2553f8a44ee837bcd431341b7e5. Latest consensus state: current_term: 1 leader_uuid: "391bb2553f8a44ee837bcd431341b7e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb5a7292e2248d28331461c809f2971" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 42881 } } peers { permanent_uuid: "ff059b5728f94491b9e2d7c8a97c641e" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38205 } } peers { permanent_uuid: "391bb2553f8a44ee837bcd431341b7e5" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45137 } } }
I20260619 14:32:08.128429 4310 tablet_replica.cc:442] T ac9bb167dc6e42d98725c8eacec0cd51 P 5bb5a7292e2248d28331461c809f2971: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:32:08.130870 4309 tablet_replica.cc:439] T ac9bb167dc6e42d98725c8eacec0cd51 P 391bb2553f8a44ee837bcd431341b7e5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "391bb2553f8a44ee837bcd431341b7e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb5a7292e2248d28331461c809f2971" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 42881 } } peers { permanent_uuid: "ff059b5728f94491b9e2d7c8a97c641e" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38205 } } peers { permanent_uuid: "391bb2553f8a44ee837bcd431341b7e5" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45137 } } }
I20260619 14:32:08.130985 4309 tablet_replica.cc:442] T ac9bb167dc6e42d98725c8eacec0cd51 P 391bb2553f8a44ee837bcd431341b7e5: This TxnStatusTablet replica's current role is: LEADER
I20260619 14:32:08.131807 4308 tablet_replica.cc:439] T ac9bb167dc6e42d98725c8eacec0cd51 P ff059b5728f94491b9e2d7c8a97c641e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "391bb2553f8a44ee837bcd431341b7e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb5a7292e2248d28331461c809f2971" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 42881 } } peers { permanent_uuid: "ff059b5728f94491b9e2d7c8a97c641e" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38205 } } peers { permanent_uuid: "391bb2553f8a44ee837bcd431341b7e5" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45137 } } }
I20260619 14:32:08.131912 4308 tablet_replica.cc:442] T ac9bb167dc6e42d98725c8eacec0cd51 P ff059b5728f94491b9e2d7c8a97c641e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:32:08.132730 4330 mvcc.cc:205] Tried to move back new op lower bound from 7298578547182563328 to 7298578547031724032. Current Snapshot: MvccSnapshot[applied={T|T < 7298578547182563328}]
I20260619 14:32:08.137349 4324 tablet_replica.cc:439] T ac9bb167dc6e42d98725c8eacec0cd51 P 391bb2553f8a44ee837bcd431341b7e5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "391bb2553f8a44ee837bcd431341b7e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb5a7292e2248d28331461c809f2971" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 42881 } } peers { permanent_uuid: "ff059b5728f94491b9e2d7c8a97c641e" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38205 } } peers { permanent_uuid: "391bb2553f8a44ee837bcd431341b7e5" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45137 } } }
I20260619 14:32:08.137476 4324 tablet_replica.cc:442] T ac9bb167dc6e42d98725c8eacec0cd51 P 391bb2553f8a44ee837bcd431341b7e5: This TxnStatusTablet replica's current role is: LEADER
I20260619 14:32:08.138657 4310 tablet_replica.cc:439] T ac9bb167dc6e42d98725c8eacec0cd51 P 5bb5a7292e2248d28331461c809f2971: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "391bb2553f8a44ee837bcd431341b7e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb5a7292e2248d28331461c809f2971" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 42881 } } peers { permanent_uuid: "ff059b5728f94491b9e2d7c8a97c641e" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38205 } } peers { permanent_uuid: "391bb2553f8a44ee837bcd431341b7e5" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45137 } } }
I20260619 14:32:08.138792 4310 tablet_replica.cc:442] T ac9bb167dc6e42d98725c8eacec0cd51 P 5bb5a7292e2248d28331461c809f2971: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:32:08.141718 4297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:08.136701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55996 (local address 127.2.2.2:38205)
0619 14:32:08.136826 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:08.136831 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:08.136845 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:32:08.136916 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:08.136919 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:08.136986 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:08.137122 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:32:08.137811 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:08.138550 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:08.139713 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:08.140005 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:08.140129 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:08.140454 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:32:08.140983 (+ 529us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:32:08.141091 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:08.141494 (+ 403us) server_negotiation.cc:299] Negotiation successful
0619 14:32:08.141547 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 14:32:08.153357 4301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:08.149532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:56711 (local address 127.2.2.3:45137)
0619 14:32:08.149670 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:08.149675 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:08.149852 (+ 177us) server_negotiation.cc:407] Connection header received
0619 14:32:08.150004 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:08.150006 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:08.150051 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:08.150181 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:32:08.150482 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:08.151117 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:08.152350 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:08.152553 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:08.152626 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:08.152983 (+ 357us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:32:08.153047 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:32:08.153098 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:08.153212 (+ 114us) server_negotiation.cc:299] Negotiation successful
0619 14:32:08.153253 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 14:32:08.153434 4340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:08.149435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.3:45137 (local address 127.2.2.2:56711)
0619 14:32:08.149761 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:08.149777 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:08.149906 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:08.150164 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:08.150167 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:08.150175 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:32:08.150360 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:08.150367 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:08.151226 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:08.151229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:08.152233 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:08.152244 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:08.152641 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:08.152679 (+ 38us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:08.152775 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:08.152828 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:32:08.153123 (+ 295us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:32:08.153129 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 14:32:08.153195 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 14:32:08.153273 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":138,"threads_started":1}
I20260619 14:32:08.160516 3953 raft_consensus.cc:1261] T 45983e43583d4fa99cc85852d3151213 P 5bb5a7292e2248d28331461c809f2971 [term 1 FOLLOWER]: Refusing update from remote peer ff059b5728f94491b9e2d7c8a97c641e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:32:08.160560 4233 raft_consensus.cc:1261] T 45983e43583d4fa99cc85852d3151213 P 391bb2553f8a44ee837bcd431341b7e5 [term 1 FOLLOWER]: Refusing update from remote peer ff059b5728f94491b9e2d7c8a97c641e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:32:08.161032 4312 consensus_queue.cc:1035] T 45983e43583d4fa99cc85852d3151213 P ff059b5728f94491b9e2d7c8a97c641e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bb5a7292e2248d28331461c809f2971" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 42881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:32:08.161235 4308 consensus_queue.cc:1035] T 45983e43583d4fa99cc85852d3151213 P ff059b5728f94491b9e2d7c8a97c641e [LEADER]: Connected to new peer: Peer: permanent_uuid: "391bb2553f8a44ee837bcd431341b7e5" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:32:08.167248 4329 mvcc.cc:205] Tried to move back new op lower bound from 7298578547340480512 to 7298578545832206336. Current Snapshot: MvccSnapshot[applied={T|T < 7298578547340480512}]
I20260619 14:32:08.170903 4331 mvcc.cc:205] Tried to move back new op lower bound from 7298578547340480512 to 7298578545832206336. Current Snapshot: MvccSnapshot[applied={T|T < 7298578547340480512}]
I20260619 14:32:08.186882 4297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:08.182696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.3:48267 (local address 127.2.2.2:38205)
0619 14:32:08.182803 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:08.182807 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:08.182885 (+ 78us) server_negotiation.cc:407] Connection header received
0619 14:32:08.183042 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:08.183045 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:08.183096 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:08.183175 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:32:08.183482 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:08.184287 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:08.185586 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:08.185957 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:08.186075 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:08.186406 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:32:08.186497 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:32:08.186580 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:08.186678 (+ 98us) server_negotiation.cc:299] Negotiation successful
0619 14:32:08.186740 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 14:32:08.186913 4345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:08.182343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.2:38205 (local address 127.2.2.3:48267)
0619 14:32:08.182818 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:08.182832 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:08.182936 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:08.183202 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:08.183204 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:08.183212 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:32:08.183347 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:08.183353 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:08.184428 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:08.184433 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:08.185444 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:08.185455 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:08.186048 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:08.186067 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:08.186206 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:08.186264 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:32:08.186596 (+ 332us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:32:08.186602 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 14:32:08.186667 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 14:32:08.186750 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":105,"threads_started":1}
I20260619 14:32:08.219942 3794 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:32:08.220170 3794 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:32:08.223142 3794 catalog_manager.cc:5782] T 00000000000000000000000000000000 P de380797462143cc92218c66aee9846e: Sending DeleteTablet for 3 replicas of tablet 45983e43583d4fa99cc85852d3151213
I20260619 14:32:08.224185 4213 tablet_service.cc:1514] Processing DeleteTablet for tablet 45983e43583d4fa99cc85852d3151213 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:32:08 UTC) from {username='kudu', principal='kudu/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:52688
I20260619 14:32:08.224186 4073 tablet_service.cc:1514] Processing DeleteTablet for tablet 45983e43583d4fa99cc85852d3151213 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:32:08 UTC) from {username='kudu', principal='kudu/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:55986
I20260619 14:32:08.224892 4353 tablet_replica.cc:330] T 45983e43583d4fa99cc85852d3151213 P 391bb2553f8a44ee837bcd431341b7e5: stopping tablet replica
I20260619 14:32:08.225045 3933 tablet_service.cc:1514] Processing DeleteTablet for tablet 45983e43583d4fa99cc85852d3151213 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:32:08 UTC) from {username='kudu', principal='kudu/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:48050
I20260619 14:32:08.225327 4353 raft_consensus.cc:2229] T 45983e43583d4fa99cc85852d3151213 P 391bb2553f8a44ee837bcd431341b7e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:32:08.225576 4353 raft_consensus.cc:2258] T 45983e43583d4fa99cc85852d3151213 P 391bb2553f8a44ee837bcd431341b7e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:32:08.226328 3794 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59818:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:32:08.226824 3794 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:32:08.227377 4353 ts_tablet_manager.cc:1905] T 45983e43583d4fa99cc85852d3151213 P 391bb2553f8a44ee837bcd431341b7e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:32:08.231556 4353 ts_tablet_manager.cc:1918] T 45983e43583d4fa99cc85852d3151213 P 391bb2553f8a44ee837bcd431341b7e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:32:08.231770 4353 log.cc:1196] T 45983e43583d4fa99cc85852d3151213 P 391bb2553f8a44ee837bcd431341b7e5: Deleting WAL directory at /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/ts-2/wal/wals/45983e43583d4fa99cc85852d3151213
I20260619 14:32:08.232270 4353 ts_tablet_manager.cc:1939] T 45983e43583d4fa99cc85852d3151213 P 391bb2553f8a44ee837bcd431341b7e5: Deleting consensus metadata
I20260619 14:32:08.232285 4213 tablet_service.cc:1467] Processing CreateTablet for tablet 40240cf585e84cfeb7f18e2e908be477 (DEFAULT_TABLE table=test-table [id=bd526b5f069146f7b63f301d5ee158af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:32:08.232666 4213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40240cf585e84cfeb7f18e2e908be477. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:08.232648 4073 tablet_service.cc:1467] Processing CreateTablet for tablet 40240cf585e84cfeb7f18e2e908be477 (DEFAULT_TABLE table=test-table [id=bd526b5f069146f7b63f301d5ee158af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:32:08.232889 4073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40240cf585e84cfeb7f18e2e908be477. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:08.235121 4303 tablet_bootstrap.cc:492] T 40240cf585e84cfeb7f18e2e908be477 P 391bb2553f8a44ee837bcd431341b7e5: Bootstrap starting.
I20260619 14:32:08.235685 3933 tablet_service.cc:1467] Processing CreateTablet for tablet 40240cf585e84cfeb7f18e2e908be477 (DEFAULT_TABLE table=test-table [id=bd526b5f069146f7b63f301d5ee158af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:32:08.235932 3933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40240cf585e84cfeb7f18e2e908be477. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:08.236133 4303 tablet_bootstrap.cc:654] T 40240cf585e84cfeb7f18e2e908be477 P 391bb2553f8a44ee837bcd431341b7e5: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:08.238222 4302 tablet_bootstrap.cc:492] T 40240cf585e84cfeb7f18e2e908be477 P 5bb5a7292e2248d28331461c809f2971: Bootstrap starting.
I20260619 14:32:08.238556 3783 catalog_manager.cc:4841] TS 391bb2553f8a44ee837bcd431341b7e5 (127.2.2.3:45137): tablet 45983e43583d4fa99cc85852d3151213 (table test-table [id=916504edf5bd492399ab146cb415e02c]) successfully deleted
I20260619 14:32:08.239059 4302 tablet_bootstrap.cc:654] T 40240cf585e84cfeb7f18e2e908be477 P 5bb5a7292e2248d28331461c809f2971: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:08.239657 4303 tablet_bootstrap.cc:492] T 40240cf585e84cfeb7f18e2e908be477 P 391bb2553f8a44ee837bcd431341b7e5: No bootstrap required, opened a new log
I20260619 14:32:08.239745 4303 ts_tablet_manager.cc:1397] T 40240cf585e84cfeb7f18e2e908be477 P 391bb2553f8a44ee837bcd431341b7e5: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260619 14:32:08.239957 4304 tablet_bootstrap.cc:492] T 40240cf585e84cfeb7f18e2e908be477 P ff059b5728f94491b9e2d7c8a97c641e: Bootstrap starting.
I20260619 14:32:08.240202 4303 raft_consensus.cc:348] T 40240cf585e84cfeb7f18e2e908be477 P 391bb2553f8a44ee837bcd431341b7e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb5a7292e2248d28331461c809f2971" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 42881 } } peers { permanent_uuid: "ff059b5728f94491b9e2d7c8a97c641e" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38205 } } peers { permanent_uuid: "391bb2553f8a44ee837bcd431341b7e5" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45137 } }
I20260619 14:32:08.240321 4303 raft_consensus.cc:374] T 40240cf585e84cfeb7f18e2e908be477 P 391bb2553f8a44ee837bcd431341b7e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:08.240365 4303 raft_consensus.cc:729] T 40240cf585e84cfeb7f18e2e908be477 P 391bb2553f8a44ee837bcd431341b7e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 391bb2553f8a44ee837bcd431341b7e5, State: Initialized, Role: FOLLOWER
I20260619 14:32:08.240489 4303 consensus_queue.cc:260] T 40240cf585e84cfeb7f18e2e908be477 P 391bb2553f8a44ee837bcd431341b7e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb5a7292e2248d28331461c809f2971" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 42881 } } peers { permanent_uuid: "ff059b5728f94491b9e2d7c8a97c641e" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38205 } } peers { permanent_uuid: "391bb2553f8a44ee837bcd431341b7e5" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45137 } }
I20260619 14:32:08.240757 4303 ts_tablet_manager.cc:1428] T 40240cf585e84cfeb7f18e2e908be477 P 391bb2553f8a44ee837bcd431341b7e5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:32:08.240903 4304 tablet_bootstrap.cc:654] T 40240cf585e84cfeb7f18e2e908be477 P ff059b5728f94491b9e2d7c8a97c641e: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:08.243755 4302 tablet_bootstrap.cc:492] T 40240cf585e84cfeb7f18e2e908be477 P 5bb5a7292e2248d28331461c809f2971: No bootstrap required, opened a new log
I20260619 14:32:08.243862 4302 ts_tablet_manager.cc:1397] T 40240cf585e84cfeb7f18e2e908be477 P 5bb5a7292e2248d28331461c809f2971: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20260619 14:32:08.243897 4304 tablet_bootstrap.cc:492] T 40240cf585e84cfeb7f18e2e908be477 P ff059b5728f94491b9e2d7c8a97c641e: No bootstrap required, opened a new log
I20260619 14:32:08.243970 4304 ts_tablet_manager.cc:1397] T 40240cf585e84cfeb7f18e2e908be477 P ff059b5728f94491b9e2d7c8a97c641e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 14:32:08.244308 4302 raft_consensus.cc:348] T 40240cf585e84cfeb7f18e2e908be477 P 5bb5a7292e2248d28331461c809f2971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb5a7292e2248d28331461c809f2971" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 42881 } } peers { permanent_uuid: "ff059b5728f94491b9e2d7c8a97c641e" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38205 } } peers { permanent_uuid: "391bb2553f8a44ee837bcd431341b7e5" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45137 } }
I20260619 14:32:08.244423 4302 raft_consensus.cc:374] T 40240cf585e84cfeb7f18e2e908be477 P 5bb5a7292e2248d28331461c809f2971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:08.244459 4302 raft_consensus.cc:729] T 40240cf585e84cfeb7f18e2e908be477 P 5bb5a7292e2248d28331461c809f2971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bb5a7292e2248d28331461c809f2971, State: Initialized, Role: FOLLOWER
I20260619 14:32:08.244408 4304 raft_consensus.cc:348] T 40240cf585e84cfeb7f18e2e908be477 P ff059b5728f94491b9e2d7c8a97c641e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb5a7292e2248d28331461c809f2971" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 42881 } } peers { permanent_uuid: "ff059b5728f94491b9e2d7c8a97c641e" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38205 } } peers { permanent_uuid: "391bb2553f8a44ee837bcd431341b7e5" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45137 } }
I20260619 14:32:08.244515 4304 raft_consensus.cc:374] T 40240cf585e84cfeb7f18e2e908be477 P ff059b5728f94491b9e2d7c8a97c641e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:08.244546 4304 raft_consensus.cc:729] T 40240cf585e84cfeb7f18e2e908be477 P ff059b5728f94491b9e2d7c8a97c641e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff059b5728f94491b9e2d7c8a97c641e, State: Initialized, Role: FOLLOWER
I20260619 14:32:08.244619 4302 consensus_queue.cc:260] T 40240cf585e84cfeb7f18e2e908be477 P 5bb5a7292e2248d28331461c809f2971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb5a7292e2248d28331461c809f2971" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 42881 } } peers { permanent_uuid: "ff059b5728f94491b9e2d7c8a97c641e" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38205 } } peers { permanent_uuid: "391bb2553f8a44ee837bcd431341b7e5" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45137 } }
I20260619 14:32:08.244686 4304 consensus_queue.cc:260] T 40240cf585e84cfeb7f18e2e908be477 P ff059b5728f94491b9e2d7c8a97c641e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb5a7292e2248d28331461c809f2971" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 42881 } } peers { permanent_uuid: "ff059b5728f94491b9e2d7c8a97c641e" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38205 } } peers { permanent_uuid: "391bb2553f8a44ee837bcd431341b7e5" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45137 } }
I20260619 14:32:08.244902 4302 ts_tablet_manager.cc:1428] T 40240cf585e84cfeb7f18e2e908be477 P 5bb5a7292e2248d28331461c809f2971: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:32:08.244939 4304 ts_tablet_manager.cc:1428] T 40240cf585e84cfeb7f18e2e908be477 P ff059b5728f94491b9e2d7c8a97c641e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:32:08.246621 4354 tablet_replica.cc:330] T 45983e43583d4fa99cc85852d3151213 P 5bb5a7292e2248d28331461c809f2971: stopping tablet replica
I20260619 14:32:08.247110 4354 raft_consensus.cc:2229] T 45983e43583d4fa99cc85852d3151213 P 5bb5a7292e2248d28331461c809f2971 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:32:08.247347 4354 raft_consensus.cc:2258] T 45983e43583d4fa99cc85852d3151213 P 5bb5a7292e2248d28331461c809f2971 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:32:08.247696 4352 tablet_replica.cc:330] T 45983e43583d4fa99cc85852d3151213 P ff059b5728f94491b9e2d7c8a97c641e: stopping tablet replica
I20260619 14:32:08.249748 4354 ts_tablet_manager.cc:1905] T 45983e43583d4fa99cc85852d3151213 P 5bb5a7292e2248d28331461c809f2971: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:32:08.247993 4352 raft_consensus.cc:2229] T 45983e43583d4fa99cc85852d3151213 P ff059b5728f94491b9e2d7c8a97c641e [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:32:08.251798 4352 raft_consensus.cc:2258] T 45983e43583d4fa99cc85852d3151213 P ff059b5728f94491b9e2d7c8a97c641e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:32:08.253819 4354 ts_tablet_manager.cc:1918] T 45983e43583d4fa99cc85852d3151213 P 5bb5a7292e2248d28331461c809f2971: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:32:08.253930 4354 log.cc:1196] T 45983e43583d4fa99cc85852d3151213 P 5bb5a7292e2248d28331461c809f2971: Deleting WAL directory at /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/ts-0/wal/wals/45983e43583d4fa99cc85852d3151213
I20260619 14:32:08.254293 4354 ts_tablet_manager.cc:1939] T 45983e43583d4fa99cc85852d3151213 P 5bb5a7292e2248d28331461c809f2971: Deleting consensus metadata
I20260619 14:32:08.255652 3781 catalog_manager.cc:4841] TS 5bb5a7292e2248d28331461c809f2971 (127.2.2.1:42881): tablet 45983e43583d4fa99cc85852d3151213 (table test-table [id=916504edf5bd492399ab146cb415e02c]) successfully deleted
I20260619 14:32:08.256536 4352 ts_tablet_manager.cc:1905] T 45983e43583d4fa99cc85852d3151213 P ff059b5728f94491b9e2d7c8a97c641e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:32:08.259632 4352 ts_tablet_manager.cc:1918] T 45983e43583d4fa99cc85852d3151213 P ff059b5728f94491b9e2d7c8a97c641e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:32:08.259743 4352 log.cc:1196] T 45983e43583d4fa99cc85852d3151213 P ff059b5728f94491b9e2d7c8a97c641e: Deleting WAL directory at /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/ts-1/wal/wals/45983e43583d4fa99cc85852d3151213
I20260619 14:32:08.260051 4352 ts_tablet_manager.cc:1939] T 45983e43583d4fa99cc85852d3151213 P ff059b5728f94491b9e2d7c8a97c641e: Deleting consensus metadata
I20260619 14:32:08.261289 3782 catalog_manager.cc:4841] TS ff059b5728f94491b9e2d7c8a97c641e (127.2.2.2:38205): tablet 45983e43583d4fa99cc85852d3151213 (table test-table [id=916504edf5bd492399ab146cb415e02c]) successfully deleted
I20260619 14:32:08.326464 4310 raft_consensus.cc:482] T 40240cf585e84cfeb7f18e2e908be477 P 5bb5a7292e2248d28331461c809f2971 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:32:08.326663 4310 raft_consensus.cc:504] T 40240cf585e84cfeb7f18e2e908be477 P 5bb5a7292e2248d28331461c809f2971 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb5a7292e2248d28331461c809f2971" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 42881 } } peers { permanent_uuid: "ff059b5728f94491b9e2d7c8a97c641e" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38205 } } peers { permanent_uuid: "391bb2553f8a44ee837bcd431341b7e5" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45137 } }
I20260619 14:32:08.327831 4310 leader_election.cc:288] T 40240cf585e84cfeb7f18e2e908be477 P 5bb5a7292e2248d28331461c809f2971 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff059b5728f94491b9e2d7c8a97c641e (127.2.2.2:38205), 391bb2553f8a44ee837bcd431341b7e5 (127.2.2.3:45137)
I20260619 14:32:08.333082 4359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:08.328074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.3:45137 (local address 127.2.2.1:43045)
0619 14:32:08.328806 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:08.328821 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:08.328909 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:08.329197 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:08.329199 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:08.329217 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:32:08.329335 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:08.329341 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:08.330314 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:08.330317 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:08.331655 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:08.331664 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:08.332588 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:08.332592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:08.332704 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:08.332735 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 14:32:08.332825 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 14:32:08.332900 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"mutex_wait_us":871,"thread_start_us":131,"threads_started":1}
I20260619 14:32:08.333112 4301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:08.328392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:43045 (local address 127.2.2.3:45137)
0619 14:32:08.328518 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:08.328524 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:08.328878 (+ 354us) server_negotiation.cc:407] Connection header received
0619 14:32:08.329046 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:08.329050 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:08.329097 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:08.329170 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:32:08.329455 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:08.330200 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:08.331793 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:08.332435 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:08.332547 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:08.332628 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:08.332854 (+ 226us) server_negotiation.cc:299] Negotiation successful
0619 14:32:08.332938 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 14:32:08.333215 4297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:08.328261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:60125 (local address 127.2.2.2:38205)
0619 14:32:08.328405 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:08.328412 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:08.328434 (+ 22us) server_negotiation.cc:407] Connection header received
0619 14:32:08.328617 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:08.328620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:08.328667 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:08.328745 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:32:08.329103 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:08.329908 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:08.331964 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:08.332561 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:08.332655 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:08.332706 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:08.333029 (+ 323us) server_negotiation.cc:299] Negotiation successful
0619 14:32:08.333101 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 14:32:08.333585 4233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40240cf585e84cfeb7f18e2e908be477" candidate_uuid: "5bb5a7292e2248d28331461c809f2971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "391bb2553f8a44ee837bcd431341b7e5" is_pre_election: true
I20260619 14:32:08.333756 4233 raft_consensus.cc:2454] T 40240cf585e84cfeb7f18e2e908be477 P 391bb2553f8a44ee837bcd431341b7e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bb5a7292e2248d28331461c809f2971 in term 0.
I20260619 14:32:08.334013 4358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:08.328074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.2:38205 (local address 127.2.2.1:60125)
0619 14:32:08.328334 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:08.328349 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:08.328456 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:08.328794 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:08.328798 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:08.328819 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:32:08.328955 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:08.328962 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:08.330023 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:08.330027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:08.331833 (+ 1806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:08.331844 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:08.332766 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:08.332769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:08.332886 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:08.332908 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 14:32:08.333744 (+ 836us) client_negotiation.cc:240] Negotiation successful
0619 14:32:08.333845 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":1,"thread_start_us":75,"threads_started":1}
I20260619 14:32:08.334151 3887 leader_election.cc:302] T 40240cf585e84cfeb7f18e2e908be477 P 5bb5a7292e2248d28331461c809f2971 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 391bb2553f8a44ee837bcd431341b7e5, 5bb5a7292e2248d28331461c809f2971; no voters:
I20260619 14:32:08.334475 4310 raft_consensus.cc:2781] T 40240cf585e84cfeb7f18e2e908be477 P 5bb5a7292e2248d28331461c809f2971 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:32:08.334550 4310 raft_consensus.cc:482] T 40240cf585e84cfeb7f18e2e908be477 P 5bb5a7292e2248d28331461c809f2971 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:32:08.334592 4310 raft_consensus.cc:3037] T 40240cf585e84cfeb7f18e2e908be477 P 5bb5a7292e2248d28331461c809f2971 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:08.334846 4093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40240cf585e84cfeb7f18e2e908be477" candidate_uuid: "5bb5a7292e2248d28331461c809f2971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff059b5728f94491b9e2d7c8a97c641e" is_pre_election: true
I20260619 14:32:08.334995 4093 raft_consensus.cc:2454] T 40240cf585e84cfeb7f18e2e908be477 P ff059b5728f94491b9e2d7c8a97c641e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bb5a7292e2248d28331461c809f2971 in term 0.
I20260619 14:32:08.336059 4310 raft_consensus.cc:504] T 40240cf585e84cfeb7f18e2e908be477 P 5bb5a7292e2248d28331461c809f2971 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb5a7292e2248d28331461c809f2971" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 42881 } } peers { permanent_uuid: "ff059b5728f94491b9e2d7c8a97c641e" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38205 } } peers { permanent_uuid: "391bb2553f8a44ee837bcd431341b7e5" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45137 } }
I20260619 14:32:08.336444 4310 leader_election.cc:288] T 40240cf585e84cfeb7f18e2e908be477 P 5bb5a7292e2248d28331461c809f2971 [CANDIDATE]: Term 1 election: Requested vote from peers ff059b5728f94491b9e2d7c8a97c641e (127.2.2.2:38205), 391bb2553f8a44ee837bcd431341b7e5 (127.2.2.3:45137)
I20260619 14:32:08.336805 4093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40240cf585e84cfeb7f18e2e908be477" candidate_uuid: "5bb5a7292e2248d28331461c809f2971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff059b5728f94491b9e2d7c8a97c641e"
I20260619 14:32:08.336830 4233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40240cf585e84cfeb7f18e2e908be477" candidate_uuid: "5bb5a7292e2248d28331461c809f2971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "391bb2553f8a44ee837bcd431341b7e5"
I20260619 14:32:08.336925 4093 raft_consensus.cc:3037] T 40240cf585e84cfeb7f18e2e908be477 P ff059b5728f94491b9e2d7c8a97c641e [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:08.336966 4233 raft_consensus.cc:3037] T 40240cf585e84cfeb7f18e2e908be477 P 391bb2553f8a44ee837bcd431341b7e5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:08.337862 4233 raft_consensus.cc:2454] T 40240cf585e84cfeb7f18e2e908be477 P 391bb2553f8a44ee837bcd431341b7e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bb5a7292e2248d28331461c809f2971 in term 1.
I20260619 14:32:08.338074 4093 raft_consensus.cc:2454] T 40240cf585e84cfeb7f18e2e908be477 P ff059b5728f94491b9e2d7c8a97c641e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bb5a7292e2248d28331461c809f2971 in term 1.
I20260619 14:32:08.338163 3887 leader_election.cc:302] T 40240cf585e84cfeb7f18e2e908be477 P 5bb5a7292e2248d28331461c809f2971 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 391bb2553f8a44ee837bcd431341b7e5, 5bb5a7292e2248d28331461c809f2971; no voters:
I20260619 14:32:08.338418 4310 raft_consensus.cc:2781] T 40240cf585e84cfeb7f18e2e908be477 P 5bb5a7292e2248d28331461c809f2971 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:32:08.338680 4310 raft_consensus.cc:686] T 40240cf585e84cfeb7f18e2e908be477 P 5bb5a7292e2248d28331461c809f2971 [term 1 LEADER]: Becoming Leader. State: Replica: 5bb5a7292e2248d28331461c809f2971, State: Running, Role: LEADER
I20260619 14:32:08.338927 4310 consensus_queue.cc:237] T 40240cf585e84cfeb7f18e2e908be477 P 5bb5a7292e2248d28331461c809f2971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb5a7292e2248d28331461c809f2971" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 42881 } } peers { permanent_uuid: "ff059b5728f94491b9e2d7c8a97c641e" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38205 } } peers { permanent_uuid: "391bb2553f8a44ee837bcd431341b7e5" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45137 } }
I20260619 14:32:08.341624 3796 catalog_manager.cc:5495] T 40240cf585e84cfeb7f18e2e908be477 P 5bb5a7292e2248d28331461c809f2971 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5bb5a7292e2248d28331461c809f2971 (127.2.2.1). New cstate: current_term: 1 leader_uuid: "5bb5a7292e2248d28331461c809f2971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb5a7292e2248d28331461c809f2971" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 42881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff059b5728f94491b9e2d7c8a97c641e" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "391bb2553f8a44ee837bcd431341b7e5" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45137 } health_report { overall_health: UNKNOWN } } }
I20260619 14:32:08.358656 4300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:08.354752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48052 (local address 127.2.2.1:42881)
0619 14:32:08.354882 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:08.354887 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:08.354903 (+ 16us) server_negotiation.cc:407] Connection header received
0619 14:32:08.355028 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:08.355032 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:08.355102 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:08.355233 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:32:08.355466 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:08.356223 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:08.357080 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:08.357310 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:08.357418 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:08.357708 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:32:08.358251 (+ 543us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:32:08.358380 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:08.358440 (+ 60us) server_negotiation.cc:299] Negotiation successful
0619 14:32:08.358498 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 14:32:08.370410 4301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:08.366411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:59821 (local address 127.2.2.3:45137)
0619 14:32:08.366562 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:08.366567 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:08.366883 (+ 316us) server_negotiation.cc:407] Connection header received
0619 14:32:08.367011 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:08.367013 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:08.367060 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:08.367145 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:32:08.367443 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:08.367937 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:08.369156 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:08.369380 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:08.369475 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:08.369845 (+ 370us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:32:08.370002 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:32:08.370093 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:08.370239 (+ 146us) server_negotiation.cc:299] Negotiation successful
0619 14:32:08.370291 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 14:32:08.370482 4364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:08.366314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.3:45137 (local address 127.2.2.1:59821)
0619 14:32:08.366768 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:08.366784 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:08.366910 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:08.367184 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:08.367187 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:08.367196 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:32:08.367336 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:08.367343 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:08.368052 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:08.368056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:08.369037 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:08.369048 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:08.369492 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:08.369514 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:08.369625 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:08.369682 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:32:08.370102 (+ 420us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:32:08.370110 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 14:32:08.370208 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 14:32:08.370293 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":119,"threads_started":1}
I20260619 14:32:08.377043 4233 raft_consensus.cc:1261] T 40240cf585e84cfeb7f18e2e908be477 P 391bb2553f8a44ee837bcd431341b7e5 [term 1 FOLLOWER]: Refusing update from remote peer 5bb5a7292e2248d28331461c809f2971: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:32:08.377269 4093 raft_consensus.cc:1261] T 40240cf585e84cfeb7f18e2e908be477 P ff059b5728f94491b9e2d7c8a97c641e [term 1 FOLLOWER]: Refusing update from remote peer 5bb5a7292e2248d28331461c809f2971: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:32:08.377751 4310 consensus_queue.cc:1035] T 40240cf585e84cfeb7f18e2e908be477 P 5bb5a7292e2248d28331461c809f2971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "391bb2553f8a44ee837bcd431341b7e5" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:32:08.377969 4360 consensus_queue.cc:1035] T 40240cf585e84cfeb7f18e2e908be477 P 5bb5a7292e2248d28331461c809f2971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff059b5728f94491b9e2d7c8a97c641e" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:32:08.398078 4300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:08.394644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.3:57211 (local address 127.2.2.1:42881)
0619 14:32:08.394757 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:08.394763 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:08.394775 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:32:08.394813 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:08.394815 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:08.394864 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:08.394949 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:32:08.395221 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:08.395773 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:08.396886 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:08.397253 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:08.397357 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:08.397616 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:32:08.397700 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:32:08.397766 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:08.397885 (+ 119us) server_negotiation.cc:299] Negotiation successful
0619 14:32:08.397940 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 14:32:08.398105 4345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:08.394373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.1:42881 (local address 127.2.2.3:57211)
0619 14:32:08.394571 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:08.394589 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:08.394676 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:08.394959 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:08.394961 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:08.394968 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:32:08.395074 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:08.395080 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:08.395888 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:08.395893 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:08.396759 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:08.396771 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:08.397362 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:08.397378 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:08.397462 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:08.397496 (+ 34us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:32:08.397786 (+ 290us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:32:08.397792 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 14:32:08.397858 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 14:32:08.397944 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 14:32:08.403069 4331 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 14:32:09.968916 3796 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:32:09.969089 3796 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:32:09.971763 3796 catalog_manager.cc:5782] T 00000000000000000000000000000000 P de380797462143cc92218c66aee9846e: Sending DeleteTablet for 3 replicas of tablet 40240cf585e84cfeb7f18e2e908be477
I20260619 14:32:09.972378 3933 tablet_service.cc:1514] Processing DeleteTablet for tablet 40240cf585e84cfeb7f18e2e908be477 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:32:09 UTC) from {username='kudu', principal='kudu/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:48050
I20260619 14:32:09.972621 4073 tablet_service.cc:1514] Processing DeleteTablet for tablet 40240cf585e84cfeb7f18e2e908be477 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:32:09 UTC) from {username='kudu', principal='kudu/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:55986
I20260619 14:32:09.972771 4390 tablet_replica.cc:330] T 40240cf585e84cfeb7f18e2e908be477 P 5bb5a7292e2248d28331461c809f2971: stopping tablet replica
I20260619 14:32:09.972918 4390 raft_consensus.cc:2229] T 40240cf585e84cfeb7f18e2e908be477 P 5bb5a7292e2248d28331461c809f2971 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:32:09.972990 4391 tablet_replica.cc:330] T 40240cf585e84cfeb7f18e2e908be477 P ff059b5728f94491b9e2d7c8a97c641e: stopping tablet replica
I20260619 14:32:09.973119 4391 raft_consensus.cc:2229] T 40240cf585e84cfeb7f18e2e908be477 P ff059b5728f94491b9e2d7c8a97c641e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:32:09.973237 4391 raft_consensus.cc:2258] T 40240cf585e84cfeb7f18e2e908be477 P ff059b5728f94491b9e2d7c8a97c641e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:32:09.973265 4390 raft_consensus.cc:2258] T 40240cf585e84cfeb7f18e2e908be477 P 5bb5a7292e2248d28331461c809f2971 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:32:09.974010 4390 ts_tablet_manager.cc:1905] T 40240cf585e84cfeb7f18e2e908be477 P 5bb5a7292e2248d28331461c809f2971: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:32:09.976038 4213 tablet_service.cc:1514] Processing DeleteTablet for tablet 40240cf585e84cfeb7f18e2e908be477 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:32:09 UTC) from {username='kudu', principal='kudu/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:52688
I20260619 14:32:09.976454 4393 tablet_replica.cc:330] T 40240cf585e84cfeb7f18e2e908be477 P 391bb2553f8a44ee837bcd431341b7e5: stopping tablet replica
I20260619 14:32:09.976675 4393 raft_consensus.cc:2229] T 40240cf585e84cfeb7f18e2e908be477 P 391bb2553f8a44ee837bcd431341b7e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:32:09.976892 4393 raft_consensus.cc:2258] T 40240cf585e84cfeb7f18e2e908be477 P 391bb2553f8a44ee837bcd431341b7e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:32:09.976961 4391 ts_tablet_manager.cc:1905] T 40240cf585e84cfeb7f18e2e908be477 P ff059b5728f94491b9e2d7c8a97c641e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:32:09.977536 4390 ts_tablet_manager.cc:1918] T 40240cf585e84cfeb7f18e2e908be477 P 5bb5a7292e2248d28331461c809f2971: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 14:32:09.977636 4390 log.cc:1196] T 40240cf585e84cfeb7f18e2e908be477 P 5bb5a7292e2248d28331461c809f2971: Deleting WAL directory at /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/ts-0/wal/wals/40240cf585e84cfeb7f18e2e908be477
I20260619 14:32:09.977823 4393 ts_tablet_manager.cc:1905] T 40240cf585e84cfeb7f18e2e908be477 P 391bb2553f8a44ee837bcd431341b7e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:32:09.977928 4390 ts_tablet_manager.cc:1939] T 40240cf585e84cfeb7f18e2e908be477 P 5bb5a7292e2248d28331461c809f2971: Deleting consensus metadata
I20260619 14:32:09.978770 3781 catalog_manager.cc:4841] TS 5bb5a7292e2248d28331461c809f2971 (127.2.2.1:42881): tablet 40240cf585e84cfeb7f18e2e908be477 (table test-table [id=bd526b5f069146f7b63f301d5ee158af]) successfully deleted
I20260619 14:32:09.979962 4391 ts_tablet_manager.cc:1918] T 40240cf585e84cfeb7f18e2e908be477 P ff059b5728f94491b9e2d7c8a97c641e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 14:32:09.980057 4391 log.cc:1196] T 40240cf585e84cfeb7f18e2e908be477 P ff059b5728f94491b9e2d7c8a97c641e: Deleting WAL directory at /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/ts-1/wal/wals/40240cf585e84cfeb7f18e2e908be477
I20260619 14:32:09.980372 4391 ts_tablet_manager.cc:1939] T 40240cf585e84cfeb7f18e2e908be477 P ff059b5728f94491b9e2d7c8a97c641e: Deleting consensus metadata
I20260619 14:32:09.981086 4393 ts_tablet_manager.cc:1918] T 40240cf585e84cfeb7f18e2e908be477 P 391bb2553f8a44ee837bcd431341b7e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 14:32:09.981204 4393 log.cc:1196] T 40240cf585e84cfeb7f18e2e908be477 P 391bb2553f8a44ee837bcd431341b7e5: Deleting WAL directory at /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/ts-2/wal/wals/40240cf585e84cfeb7f18e2e908be477
I20260619 14:32:09.981240 3782 catalog_manager.cc:4841] TS ff059b5728f94491b9e2d7c8a97c641e (127.2.2.2:38205): tablet 40240cf585e84cfeb7f18e2e908be477 (table test-table [id=bd526b5f069146f7b63f301d5ee158af]) successfully deleted
I20260619 14:32:09.981516 4393 ts_tablet_manager.cc:1939] T 40240cf585e84cfeb7f18e2e908be477 P 391bb2553f8a44ee837bcd431341b7e5: Deleting consensus metadata
I20260619 14:32:09.982256 3783 catalog_manager.cc:4841] TS 391bb2553f8a44ee837bcd431341b7e5 (127.2.2.3:45137): tablet 40240cf585e84cfeb7f18e2e908be477 (table test-table [id=bd526b5f069146f7b63f301d5ee158af]) successfully deleted
Jun 19 14:32:09 dist-test-slave-70rm krb5kdc[3751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879529, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 14:32:09 dist-test-slave-70rm krb5kdc[3751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879529, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:32:10.001657 4400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:09.991872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59830 (local address 127.2.2.62:42985)
0619 14:32:09.992192 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:09.992199 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:09.992214 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:32:09.992339 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:09.992341 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:09.992385 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:09.992469 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:09.993264 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:09.994002 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:09.995104 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:09.995355 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:09.995449 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:09.998033 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:09.998037 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:09.998039 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:09.998070 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:09.999890 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:10.000418 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:10.000422 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:10.000423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:10.000514 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:10.000763 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:10.000765 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:10.000767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:10.000954 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:10.001059 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:10.001258 (+ 199us) server_negotiation.cc:299] Negotiation successful
0619 14:32:10.001433 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":121,"threads_started":1}
I20260619 14:32:10.004801 3796 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59830:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:32:10.005266 3796 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:32:10.010418 3933 tablet_service.cc:1467] Processing CreateTablet for tablet aa201022eb034c15abd551ea07445cb2 (DEFAULT_TABLE table=test-table [id=16e11cb0c18e4ca8be2efe0f852bd2c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:32:10.010452 4213 tablet_service.cc:1467] Processing CreateTablet for tablet aa201022eb034c15abd551ea07445cb2 (DEFAULT_TABLE table=test-table [id=16e11cb0c18e4ca8be2efe0f852bd2c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:32:10.010452 4073 tablet_service.cc:1467] Processing CreateTablet for tablet aa201022eb034c15abd551ea07445cb2 (DEFAULT_TABLE table=test-table [id=16e11cb0c18e4ca8be2efe0f852bd2c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:32:10.010766 4073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa201022eb034c15abd551ea07445cb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:10.010766 4213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa201022eb034c15abd551ea07445cb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:10.010776 3933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa201022eb034c15abd551ea07445cb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:10.013036 4401 tablet_bootstrap.cc:492] T aa201022eb034c15abd551ea07445cb2 P 5bb5a7292e2248d28331461c809f2971: Bootstrap starting.
I20260619 14:32:10.013654 4403 tablet_bootstrap.cc:492] T aa201022eb034c15abd551ea07445cb2 P 391bb2553f8a44ee837bcd431341b7e5: Bootstrap starting.
I20260619 14:32:10.014007 4401 tablet_bootstrap.cc:654] T aa201022eb034c15abd551ea07445cb2 P 5bb5a7292e2248d28331461c809f2971: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:10.014510 4403 tablet_bootstrap.cc:654] T aa201022eb034c15abd551ea07445cb2 P 391bb2553f8a44ee837bcd431341b7e5: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:10.015275 4401 tablet_bootstrap.cc:492] T aa201022eb034c15abd551ea07445cb2 P 5bb5a7292e2248d28331461c809f2971: No bootstrap required, opened a new log
I20260619 14:32:10.015273 4402 tablet_bootstrap.cc:492] T aa201022eb034c15abd551ea07445cb2 P ff059b5728f94491b9e2d7c8a97c641e: Bootstrap starting.
I20260619 14:32:10.015389 4401 ts_tablet_manager.cc:1397] T aa201022eb034c15abd551ea07445cb2 P 5bb5a7292e2248d28331461c809f2971: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 14:32:10.015823 4401 raft_consensus.cc:348] T aa201022eb034c15abd551ea07445cb2 P 5bb5a7292e2248d28331461c809f2971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb5a7292e2248d28331461c809f2971" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 42881 } } peers { permanent_uuid: "ff059b5728f94491b9e2d7c8a97c641e" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38205 } } peers { permanent_uuid: "391bb2553f8a44ee837bcd431341b7e5" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45137 } }
I20260619 14:32:10.015969 4401 raft_consensus.cc:374] T aa201022eb034c15abd551ea07445cb2 P 5bb5a7292e2248d28331461c809f2971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:10.016011 4401 raft_consensus.cc:729] T aa201022eb034c15abd551ea07445cb2 P 5bb5a7292e2248d28331461c809f2971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bb5a7292e2248d28331461c809f2971, State: Initialized, Role: FOLLOWER
I20260619 14:32:10.016110 4402 tablet_bootstrap.cc:654] T aa201022eb034c15abd551ea07445cb2 P ff059b5728f94491b9e2d7c8a97c641e: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:10.016147 4401 consensus_queue.cc:260] T aa201022eb034c15abd551ea07445cb2 P 5bb5a7292e2248d28331461c809f2971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb5a7292e2248d28331461c809f2971" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 42881 } } peers { permanent_uuid: "ff059b5728f94491b9e2d7c8a97c641e" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38205 } } peers { permanent_uuid: "391bb2553f8a44ee837bcd431341b7e5" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45137 } }
I20260619 14:32:10.016460 4401 ts_tablet_manager.cc:1428] T aa201022eb034c15abd551ea07445cb2 P 5bb5a7292e2248d28331461c809f2971: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 14:32:10.017508 4403 tablet_bootstrap.cc:492] T aa201022eb034c15abd551ea07445cb2 P 391bb2553f8a44ee837bcd431341b7e5: No bootstrap required, opened a new log
I20260619 14:32:10.017612 4403 ts_tablet_manager.cc:1397] T aa201022eb034c15abd551ea07445cb2 P 391bb2553f8a44ee837bcd431341b7e5: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 14:32:10.017982 4403 raft_consensus.cc:348] T aa201022eb034c15abd551ea07445cb2 P 391bb2553f8a44ee837bcd431341b7e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb5a7292e2248d28331461c809f2971" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 42881 } } peers { permanent_uuid: "ff059b5728f94491b9e2d7c8a97c641e" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38205 } } peers { permanent_uuid: "391bb2553f8a44ee837bcd431341b7e5" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45137 } }
I20260619 14:32:10.018077 4403 raft_consensus.cc:374] T aa201022eb034c15abd551ea07445cb2 P 391bb2553f8a44ee837bcd431341b7e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:10.018100 4403 raft_consensus.cc:729] T aa201022eb034c15abd551ea07445cb2 P 391bb2553f8a44ee837bcd431341b7e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 391bb2553f8a44ee837bcd431341b7e5, State: Initialized, Role: FOLLOWER
I20260619 14:32:10.018303 4403 consensus_queue.cc:260] T aa201022eb034c15abd551ea07445cb2 P 391bb2553f8a44ee837bcd431341b7e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb5a7292e2248d28331461c809f2971" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 42881 } } peers { permanent_uuid: "ff059b5728f94491b9e2d7c8a97c641e" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38205 } } peers { permanent_uuid: "391bb2553f8a44ee837bcd431341b7e5" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45137 } }
I20260619 14:32:10.018646 4403 ts_tablet_manager.cc:1428] T aa201022eb034c15abd551ea07445cb2 P 391bb2553f8a44ee837bcd431341b7e5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:32:10.019207 4402 tablet_bootstrap.cc:492] T aa201022eb034c15abd551ea07445cb2 P ff059b5728f94491b9e2d7c8a97c641e: No bootstrap required, opened a new log
I20260619 14:32:10.019321 4402 ts_tablet_manager.cc:1397] T aa201022eb034c15abd551ea07445cb2 P ff059b5728f94491b9e2d7c8a97c641e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 14:32:10.019804 4402 raft_consensus.cc:348] T aa201022eb034c15abd551ea07445cb2 P ff059b5728f94491b9e2d7c8a97c641e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb5a7292e2248d28331461c809f2971" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 42881 } } peers { permanent_uuid: "ff059b5728f94491b9e2d7c8a97c641e" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38205 } } peers { permanent_uuid: "391bb2553f8a44ee837bcd431341b7e5" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45137 } }
I20260619 14:32:10.019928 4402 raft_consensus.cc:374] T aa201022eb034c15abd551ea07445cb2 P ff059b5728f94491b9e2d7c8a97c641e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:10.019974 4402 raft_consensus.cc:729] T aa201022eb034c15abd551ea07445cb2 P ff059b5728f94491b9e2d7c8a97c641e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff059b5728f94491b9e2d7c8a97c641e, State: Initialized, Role: FOLLOWER
I20260619 14:32:10.020094 4402 consensus_queue.cc:260] T aa201022eb034c15abd551ea07445cb2 P ff059b5728f94491b9e2d7c8a97c641e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb5a7292e2248d28331461c809f2971" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 42881 } } peers { permanent_uuid: "ff059b5728f94491b9e2d7c8a97c641e" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38205 } } peers { permanent_uuid: "391bb2553f8a44ee837bcd431341b7e5" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45137 } }
I20260619 14:32:10.020534 4402 ts_tablet_manager.cc:1428] T aa201022eb034c15abd551ea07445cb2 P ff059b5728f94491b9e2d7c8a97c641e: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260619 14:32:10.163703 4407 raft_consensus.cc:482] T aa201022eb034c15abd551ea07445cb2 P ff059b5728f94491b9e2d7c8a97c641e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:32:10.163869 4407 raft_consensus.cc:504] T aa201022eb034c15abd551ea07445cb2 P ff059b5728f94491b9e2d7c8a97c641e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb5a7292e2248d28331461c809f2971" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 42881 } } peers { permanent_uuid: "ff059b5728f94491b9e2d7c8a97c641e" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38205 } } peers { permanent_uuid: "391bb2553f8a44ee837bcd431341b7e5" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45137 } }
I20260619 14:32:10.164412 4407 leader_election.cc:288] T aa201022eb034c15abd551ea07445cb2 P ff059b5728f94491b9e2d7c8a97c641e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5bb5a7292e2248d28331461c809f2971 (127.2.2.1:42881), 391bb2553f8a44ee837bcd431341b7e5 (127.2.2.3:45137)
I20260619 14:32:10.164793 3952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa201022eb034c15abd551ea07445cb2" candidate_uuid: "ff059b5728f94491b9e2d7c8a97c641e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bb5a7292e2248d28331461c809f2971" is_pre_election: true
I20260619 14:32:10.164896 4233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa201022eb034c15abd551ea07445cb2" candidate_uuid: "ff059b5728f94491b9e2d7c8a97c641e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "391bb2553f8a44ee837bcd431341b7e5" is_pre_election: true
I20260619 14:32:10.164965 3952 raft_consensus.cc:2454] T aa201022eb034c15abd551ea07445cb2 P 5bb5a7292e2248d28331461c809f2971 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff059b5728f94491b9e2d7c8a97c641e in term 0.
I20260619 14:32:10.165032 4233 raft_consensus.cc:2454] T aa201022eb034c15abd551ea07445cb2 P 391bb2553f8a44ee837bcd431341b7e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff059b5728f94491b9e2d7c8a97c641e in term 0.
I20260619 14:32:10.165372 4027 leader_election.cc:302] T aa201022eb034c15abd551ea07445cb2 P ff059b5728f94491b9e2d7c8a97c641e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 391bb2553f8a44ee837bcd431341b7e5, ff059b5728f94491b9e2d7c8a97c641e; no voters:
I20260619 14:32:10.165591 4407 raft_consensus.cc:2781] T aa201022eb034c15abd551ea07445cb2 P ff059b5728f94491b9e2d7c8a97c641e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:32:10.165652 4407 raft_consensus.cc:482] T aa201022eb034c15abd551ea07445cb2 P ff059b5728f94491b9e2d7c8a97c641e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:32:10.165721 4407 raft_consensus.cc:3037] T aa201022eb034c15abd551ea07445cb2 P ff059b5728f94491b9e2d7c8a97c641e [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:10.166841 4407 raft_consensus.cc:504] T aa201022eb034c15abd551ea07445cb2 P ff059b5728f94491b9e2d7c8a97c641e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb5a7292e2248d28331461c809f2971" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 42881 } } peers { permanent_uuid: "ff059b5728f94491b9e2d7c8a97c641e" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38205 } } peers { permanent_uuid: "391bb2553f8a44ee837bcd431341b7e5" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45137 } }
I20260619 14:32:10.167222 4407 leader_election.cc:288] T aa201022eb034c15abd551ea07445cb2 P ff059b5728f94491b9e2d7c8a97c641e [CANDIDATE]: Term 1 election: Requested vote from peers 5bb5a7292e2248d28331461c809f2971 (127.2.2.1:42881), 391bb2553f8a44ee837bcd431341b7e5 (127.2.2.3:45137)
I20260619 14:32:10.167583 3952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa201022eb034c15abd551ea07445cb2" candidate_uuid: "ff059b5728f94491b9e2d7c8a97c641e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bb5a7292e2248d28331461c809f2971"
I20260619 14:32:10.167595 4233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa201022eb034c15abd551ea07445cb2" candidate_uuid: "ff059b5728f94491b9e2d7c8a97c641e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "391bb2553f8a44ee837bcd431341b7e5"
I20260619 14:32:10.167697 3952 raft_consensus.cc:3037] T aa201022eb034c15abd551ea07445cb2 P 5bb5a7292e2248d28331461c809f2971 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:10.167732 4233 raft_consensus.cc:3037] T aa201022eb034c15abd551ea07445cb2 P 391bb2553f8a44ee837bcd431341b7e5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:10.168697 4233 raft_consensus.cc:2454] T aa201022eb034c15abd551ea07445cb2 P 391bb2553f8a44ee837bcd431341b7e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff059b5728f94491b9e2d7c8a97c641e in term 1.
I20260619 14:32:10.168910 3952 raft_consensus.cc:2454] T aa201022eb034c15abd551ea07445cb2 P 5bb5a7292e2248d28331461c809f2971 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff059b5728f94491b9e2d7c8a97c641e in term 1.
I20260619 14:32:10.169014 4027 leader_election.cc:302] T aa201022eb034c15abd551ea07445cb2 P ff059b5728f94491b9e2d7c8a97c641e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 391bb2553f8a44ee837bcd431341b7e5, ff059b5728f94491b9e2d7c8a97c641e; no voters:
I20260619 14:32:10.169173 4407 raft_consensus.cc:2781] T aa201022eb034c15abd551ea07445cb2 P ff059b5728f94491b9e2d7c8a97c641e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:32:10.169380 4407 raft_consensus.cc:686] T aa201022eb034c15abd551ea07445cb2 P ff059b5728f94491b9e2d7c8a97c641e [term 1 LEADER]: Becoming Leader. State: Replica: ff059b5728f94491b9e2d7c8a97c641e, State: Running, Role: LEADER
I20260619 14:32:10.169541 4407 consensus_queue.cc:237] T aa201022eb034c15abd551ea07445cb2 P ff059b5728f94491b9e2d7c8a97c641e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb5a7292e2248d28331461c809f2971" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 42881 } } peers { permanent_uuid: "ff059b5728f94491b9e2d7c8a97c641e" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38205 } } peers { permanent_uuid: "391bb2553f8a44ee837bcd431341b7e5" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45137 } }
I20260619 14:32:10.171311 3796 catalog_manager.cc:5495] T aa201022eb034c15abd551ea07445cb2 P ff059b5728f94491b9e2d7c8a97c641e reported cstate change: term changed from 0 to 1, leader changed from <none> to ff059b5728f94491b9e2d7c8a97c641e (127.2.2.2). New cstate: current_term: 1 leader_uuid: "ff059b5728f94491b9e2d7c8a97c641e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb5a7292e2248d28331461c809f2971" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 42881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff059b5728f94491b9e2d7c8a97c641e" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "391bb2553f8a44ee837bcd431341b7e5" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45137 } health_report { overall_health: UNKNOWN } } }
I20260619 14:32:10.214708 4411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:10.211377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56010 (local address 127.2.2.2:38205)
0619 14:32:10.211670 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:10.211677 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:10.211689 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:32:10.211730 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:10.211732 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:10.211771 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:10.211847 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:32:10.212128 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:10.212635 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:10.213466 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:10.213689 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:10.213770 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:10.214099 (+ 329us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:32:10.214245 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:32:10.214313 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:10.214463 (+ 150us) server_negotiation.cc:299] Negotiation successful
0619 14:32:10.214512 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":151,"threads_started":1}
I20260619 14:32:10.220986 4233 raft_consensus.cc:1261] T aa201022eb034c15abd551ea07445cb2 P 391bb2553f8a44ee837bcd431341b7e5 [term 1 FOLLOWER]: Refusing update from remote peer ff059b5728f94491b9e2d7c8a97c641e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:32:10.221424 3952 raft_consensus.cc:1261] T aa201022eb034c15abd551ea07445cb2 P 5bb5a7292e2248d28331461c809f2971 [term 1 FOLLOWER]: Refusing update from remote peer ff059b5728f94491b9e2d7c8a97c641e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:32:10.221407 4407 consensus_queue.cc:1035] T aa201022eb034c15abd551ea07445cb2 P ff059b5728f94491b9e2d7c8a97c641e [LEADER]: Connected to new peer: Peer: permanent_uuid: "391bb2553f8a44ee837bcd431341b7e5" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:32:10.222029 4407 consensus_queue.cc:1035] T aa201022eb034c15abd551ea07445cb2 P ff059b5728f94491b9e2d7c8a97c641e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bb5a7292e2248d28331461c809f2971" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 42881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:32:10.257813 3796 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:32:10.258006 3796 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:32:10.260239 3796 catalog_manager.cc:5782] T 00000000000000000000000000000000 P de380797462143cc92218c66aee9846e: Sending DeleteTablet for 3 replicas of tablet aa201022eb034c15abd551ea07445cb2
I20260619 14:32:10.260763 3933 tablet_service.cc:1514] Processing DeleteTablet for tablet aa201022eb034c15abd551ea07445cb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:32:10 UTC) from {username='kudu', principal='kudu/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:48050
I20260619 14:32:10.260890 4213 tablet_service.cc:1514] Processing DeleteTablet for tablet aa201022eb034c15abd551ea07445cb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:32:10 UTC) from {username='kudu', principal='kudu/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:52688
I20260619 14:32:10.261001 4073 tablet_service.cc:1514] Processing DeleteTablet for tablet aa201022eb034c15abd551ea07445cb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:32:10 UTC) from {username='kudu', principal='kudu/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:55986
I20260619 14:32:10.261024 4390 tablet_replica.cc:330] T aa201022eb034c15abd551ea07445cb2 P 5bb5a7292e2248d28331461c809f2971: stopping tablet replica
I20260619 14:32:10.261193 4391 tablet_replica.cc:330] T aa201022eb034c15abd551ea07445cb2 P ff059b5728f94491b9e2d7c8a97c641e: stopping tablet replica
I20260619 14:32:10.261202 4393 tablet_replica.cc:330] T aa201022eb034c15abd551ea07445cb2 P 391bb2553f8a44ee837bcd431341b7e5: stopping tablet replica
I20260619 14:32:10.261184 4390 raft_consensus.cc:2229] T aa201022eb034c15abd551ea07445cb2 P 5bb5a7292e2248d28331461c809f2971 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:32:10.261310 4390 raft_consensus.cc:2258] T aa201022eb034c15abd551ea07445cb2 P 5bb5a7292e2248d28331461c809f2971 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:32:10.261305 4391 raft_consensus.cc:2229] T aa201022eb034c15abd551ea07445cb2 P ff059b5728f94491b9e2d7c8a97c641e [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:32:10.261334 4393 raft_consensus.cc:2229] T aa201022eb034c15abd551ea07445cb2 P 391bb2553f8a44ee837bcd431341b7e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:32:10.261461 4393 raft_consensus.cc:2258] T aa201022eb034c15abd551ea07445cb2 P 391bb2553f8a44ee837bcd431341b7e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:32:10.261502 4391 raft_consensus.cc:2258] T aa201022eb034c15abd551ea07445cb2 P ff059b5728f94491b9e2d7c8a97c641e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:32:10.262099 4390 ts_tablet_manager.cc:1905] T aa201022eb034c15abd551ea07445cb2 P 5bb5a7292e2248d28331461c809f2971: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:32:10.262230 4391 ts_tablet_manager.cc:1905] T aa201022eb034c15abd551ea07445cb2 P ff059b5728f94491b9e2d7c8a97c641e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:32:10.262290 4393 ts_tablet_manager.cc:1905] T aa201022eb034c15abd551ea07445cb2 P 391bb2553f8a44ee837bcd431341b7e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:32:10.262290 3796 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59830:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:32:10.262713 3796 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:32:10.264703 4390 ts_tablet_manager.cc:1918] T aa201022eb034c15abd551ea07445cb2 P 5bb5a7292e2248d28331461c809f2971: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:32:10.264734 4391 ts_tablet_manager.cc:1918] T aa201022eb034c15abd551ea07445cb2 P ff059b5728f94491b9e2d7c8a97c641e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:32:10.264775 4390 log.cc:1196] T aa201022eb034c15abd551ea07445cb2 P 5bb5a7292e2248d28331461c809f2971: Deleting WAL directory at /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/ts-0/wal/wals/aa201022eb034c15abd551ea07445cb2
I20260619 14:32:10.264794 4391 log.cc:1196] T aa201022eb034c15abd551ea07445cb2 P ff059b5728f94491b9e2d7c8a97c641e: Deleting WAL directory at /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/ts-1/wal/wals/aa201022eb034c15abd551ea07445cb2
I20260619 14:32:10.264900 4393 ts_tablet_manager.cc:1918] T aa201022eb034c15abd551ea07445cb2 P 391bb2553f8a44ee837bcd431341b7e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:32:10.264971 4393 log.cc:1196] T aa201022eb034c15abd551ea07445cb2 P 391bb2553f8a44ee837bcd431341b7e5: Deleting WAL directory at /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/ts-2/wal/wals/aa201022eb034c15abd551ea07445cb2
I20260619 14:32:10.265048 4391 ts_tablet_manager.cc:1939] T aa201022eb034c15abd551ea07445cb2 P ff059b5728f94491b9e2d7c8a97c641e: Deleting consensus metadata
I20260619 14:32:10.265226 4393 ts_tablet_manager.cc:1939] T aa201022eb034c15abd551ea07445cb2 P 391bb2553f8a44ee837bcd431341b7e5: Deleting consensus metadata
I20260619 14:32:10.265731 3782 catalog_manager.cc:4841] TS ff059b5728f94491b9e2d7c8a97c641e (127.2.2.2:38205): tablet aa201022eb034c15abd551ea07445cb2 (table test-table [id=16e11cb0c18e4ca8be2efe0f852bd2c7]) successfully deleted
I20260619 14:32:10.265882 3783 catalog_manager.cc:4841] TS 391bb2553f8a44ee837bcd431341b7e5 (127.2.2.3:45137): tablet aa201022eb034c15abd551ea07445cb2 (table test-table [id=16e11cb0c18e4ca8be2efe0f852bd2c7]) successfully deleted
I20260619 14:32:10.266458 4390 ts_tablet_manager.cc:1939] T aa201022eb034c15abd551ea07445cb2 P 5bb5a7292e2248d28331461c809f2971: Deleting consensus metadata
I20260619 14:32:10.267133 3781 catalog_manager.cc:4841] TS 5bb5a7292e2248d28331461c809f2971 (127.2.2.1:42881): tablet aa201022eb034c15abd551ea07445cb2 (table test-table [id=16e11cb0c18e4ca8be2efe0f852bd2c7]) successfully deleted
I20260619 14:32:10.267853 3933 tablet_service.cc:1467] Processing CreateTablet for tablet 09aa77aa308a4e94aacaca359f9ba36d (DEFAULT_TABLE table=test-table [id=ab58e486446b4ce583f81d82a92d2d9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:32:10.268105 3933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09aa77aa308a4e94aacaca359f9ba36d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:10.268080 4213 tablet_service.cc:1467] Processing CreateTablet for tablet 09aa77aa308a4e94aacaca359f9ba36d (DEFAULT_TABLE table=test-table [id=ab58e486446b4ce583f81d82a92d2d9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:32:10.268292 4213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09aa77aa308a4e94aacaca359f9ba36d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:10.270531 4401 tablet_bootstrap.cc:492] T 09aa77aa308a4e94aacaca359f9ba36d P 5bb5a7292e2248d28331461c809f2971: Bootstrap starting.
I20260619 14:32:10.270577 4403 tablet_bootstrap.cc:492] T 09aa77aa308a4e94aacaca359f9ba36d P 391bb2553f8a44ee837bcd431341b7e5: Bootstrap starting.
I20260619 14:32:10.270823 4073 tablet_service.cc:1467] Processing CreateTablet for tablet 09aa77aa308a4e94aacaca359f9ba36d (DEFAULT_TABLE table=test-table [id=ab58e486446b4ce583f81d82a92d2d9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:32:10.271068 4073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09aa77aa308a4e94aacaca359f9ba36d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:10.271366 4401 tablet_bootstrap.cc:654] T 09aa77aa308a4e94aacaca359f9ba36d P 5bb5a7292e2248d28331461c809f2971: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:10.271457 4403 tablet_bootstrap.cc:654] T 09aa77aa308a4e94aacaca359f9ba36d P 391bb2553f8a44ee837bcd431341b7e5: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:10.272622 4401 tablet_bootstrap.cc:492] T 09aa77aa308a4e94aacaca359f9ba36d P 5bb5a7292e2248d28331461c809f2971: No bootstrap required, opened a new log
I20260619 14:32:10.272706 4401 ts_tablet_manager.cc:1397] T 09aa77aa308a4e94aacaca359f9ba36d P 5bb5a7292e2248d28331461c809f2971: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 14:32:10.273051 4401 raft_consensus.cc:348] T 09aa77aa308a4e94aacaca359f9ba36d P 5bb5a7292e2248d28331461c809f2971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb5a7292e2248d28331461c809f2971" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 42881 } } peers { permanent_uuid: "ff059b5728f94491b9e2d7c8a97c641e" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38205 } } peers { permanent_uuid: "391bb2553f8a44ee837bcd431341b7e5" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45137 } }
I20260619 14:32:10.273209 4402 tablet_bootstrap.cc:492] T 09aa77aa308a4e94aacaca359f9ba36d P ff059b5728f94491b9e2d7c8a97c641e: Bootstrap starting.
I20260619 14:32:10.273208 4401 raft_consensus.cc:374] T 09aa77aa308a4e94aacaca359f9ba36d P 5bb5a7292e2248d28331461c809f2971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:10.273249 4401 raft_consensus.cc:729] T 09aa77aa308a4e94aacaca359f9ba36d P 5bb5a7292e2248d28331461c809f2971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bb5a7292e2248d28331461c809f2971, State: Initialized, Role: FOLLOWER
I20260619 14:32:10.273386 4401 consensus_queue.cc:260] T 09aa77aa308a4e94aacaca359f9ba36d P 5bb5a7292e2248d28331461c809f2971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb5a7292e2248d28331461c809f2971" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 42881 } } peers { permanent_uuid: "ff059b5728f94491b9e2d7c8a97c641e" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38205 } } peers { permanent_uuid: "391bb2553f8a44ee837bcd431341b7e5" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45137 } }
I20260619 14:32:10.273698 4401 ts_tablet_manager.cc:1428] T 09aa77aa308a4e94aacaca359f9ba36d P 5bb5a7292e2248d28331461c809f2971: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:32:10.273938 4403 tablet_bootstrap.cc:492] T 09aa77aa308a4e94aacaca359f9ba36d P 391bb2553f8a44ee837bcd431341b7e5: No bootstrap required, opened a new log
I20260619 14:32:10.274044 4403 ts_tablet_manager.cc:1397] T 09aa77aa308a4e94aacaca359f9ba36d P 391bb2553f8a44ee837bcd431341b7e5: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260619 14:32:10.274230 4402 tablet_bootstrap.cc:654] T 09aa77aa308a4e94aacaca359f9ba36d P ff059b5728f94491b9e2d7c8a97c641e: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:10.274490 4403 raft_consensus.cc:348] T 09aa77aa308a4e94aacaca359f9ba36d P 391bb2553f8a44ee837bcd431341b7e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb5a7292e2248d28331461c809f2971" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 42881 } } peers { permanent_uuid: "ff059b5728f94491b9e2d7c8a97c641e" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38205 } } peers { permanent_uuid: "391bb2553f8a44ee837bcd431341b7e5" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45137 } }
I20260619 14:32:10.274614 4403 raft_consensus.cc:374] T 09aa77aa308a4e94aacaca359f9ba36d P 391bb2553f8a44ee837bcd431341b7e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:10.274652 4403 raft_consensus.cc:729] T 09aa77aa308a4e94aacaca359f9ba36d P 391bb2553f8a44ee837bcd431341b7e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 391bb2553f8a44ee837bcd431341b7e5, State: Initialized, Role: FOLLOWER
I20260619 14:32:10.274787 4403 consensus_queue.cc:260] T 09aa77aa308a4e94aacaca359f9ba36d P 391bb2553f8a44ee837bcd431341b7e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb5a7292e2248d28331461c809f2971" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 42881 } } peers { permanent_uuid: "ff059b5728f94491b9e2d7c8a97c641e" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38205 } } peers { permanent_uuid: "391bb2553f8a44ee837bcd431341b7e5" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45137 } }
I20260619 14:32:10.275115 4403 ts_tablet_manager.cc:1428] T 09aa77aa308a4e94aacaca359f9ba36d P 391bb2553f8a44ee837bcd431341b7e5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:32:10.275460 4402 tablet_bootstrap.cc:492] T 09aa77aa308a4e94aacaca359f9ba36d P ff059b5728f94491b9e2d7c8a97c641e: No bootstrap required, opened a new log
I20260619 14:32:10.275550 4402 ts_tablet_manager.cc:1397] T 09aa77aa308a4e94aacaca359f9ba36d P ff059b5728f94491b9e2d7c8a97c641e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260619 14:32:10.275921 4402 raft_consensus.cc:348] T 09aa77aa308a4e94aacaca359f9ba36d P ff059b5728f94491b9e2d7c8a97c641e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb5a7292e2248d28331461c809f2971" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 42881 } } peers { permanent_uuid: "ff059b5728f94491b9e2d7c8a97c641e" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38205 } } peers { permanent_uuid: "391bb2553f8a44ee837bcd431341b7e5" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45137 } }
I20260619 14:32:10.276001 4402 raft_consensus.cc:374] T 09aa77aa308a4e94aacaca359f9ba36d P ff059b5728f94491b9e2d7c8a97c641e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:10.276021 4402 raft_consensus.cc:729] T 09aa77aa308a4e94aacaca359f9ba36d P ff059b5728f94491b9e2d7c8a97c641e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff059b5728f94491b9e2d7c8a97c641e, State: Initialized, Role: FOLLOWER
I20260619 14:32:10.276109 4402 consensus_queue.cc:260] T 09aa77aa308a4e94aacaca359f9ba36d P ff059b5728f94491b9e2d7c8a97c641e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb5a7292e2248d28331461c809f2971" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 42881 } } peers { permanent_uuid: "ff059b5728f94491b9e2d7c8a97c641e" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38205 } } peers { permanent_uuid: "391bb2553f8a44ee837bcd431341b7e5" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45137 } }
I20260619 14:32:10.276324 4402 ts_tablet_manager.cc:1428] T 09aa77aa308a4e94aacaca359f9ba36d P ff059b5728f94491b9e2d7c8a97c641e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 14:32:10.616241 4371 raft_consensus.cc:482] T 09aa77aa308a4e94aacaca359f9ba36d P 391bb2553f8a44ee837bcd431341b7e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:32:10.616426 4371 raft_consensus.cc:504] T 09aa77aa308a4e94aacaca359f9ba36d P 391bb2553f8a44ee837bcd431341b7e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb5a7292e2248d28331461c809f2971" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 42881 } } peers { permanent_uuid: "ff059b5728f94491b9e2d7c8a97c641e" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38205 } } peers { permanent_uuid: "391bb2553f8a44ee837bcd431341b7e5" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45137 } }
I20260619 14:32:10.616881 4371 leader_election.cc:288] T 09aa77aa308a4e94aacaca359f9ba36d P 391bb2553f8a44ee837bcd431341b7e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5bb5a7292e2248d28331461c809f2971 (127.2.2.1:42881), ff059b5728f94491b9e2d7c8a97c641e (127.2.2.2:38205)
I20260619 14:32:10.617476 3952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09aa77aa308a4e94aacaca359f9ba36d" candidate_uuid: "391bb2553f8a44ee837bcd431341b7e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bb5a7292e2248d28331461c809f2971" is_pre_election: true
I20260619 14:32:10.617501 4093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09aa77aa308a4e94aacaca359f9ba36d" candidate_uuid: "391bb2553f8a44ee837bcd431341b7e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff059b5728f94491b9e2d7c8a97c641e" is_pre_election: true
I20260619 14:32:10.617659 3952 raft_consensus.cc:2454] T 09aa77aa308a4e94aacaca359f9ba36d P 5bb5a7292e2248d28331461c809f2971 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 391bb2553f8a44ee837bcd431341b7e5 in term 0.
I20260619 14:32:10.617682 4093 raft_consensus.cc:2454] T 09aa77aa308a4e94aacaca359f9ba36d P ff059b5728f94491b9e2d7c8a97c641e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 391bb2553f8a44ee837bcd431341b7e5 in term 0.
I20260619 14:32:10.618041 4165 leader_election.cc:302] T 09aa77aa308a4e94aacaca359f9ba36d P 391bb2553f8a44ee837bcd431341b7e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 391bb2553f8a44ee837bcd431341b7e5, 5bb5a7292e2248d28331461c809f2971; no voters:
I20260619 14:32:10.618247 4371 raft_consensus.cc:2781] T 09aa77aa308a4e94aacaca359f9ba36d P 391bb2553f8a44ee837bcd431341b7e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:32:10.618309 4371 raft_consensus.cc:482] T 09aa77aa308a4e94aacaca359f9ba36d P 391bb2553f8a44ee837bcd431341b7e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:32:10.618378 4371 raft_consensus.cc:3037] T 09aa77aa308a4e94aacaca359f9ba36d P 391bb2553f8a44ee837bcd431341b7e5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:10.619532 4371 raft_consensus.cc:504] T 09aa77aa308a4e94aacaca359f9ba36d P 391bb2553f8a44ee837bcd431341b7e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb5a7292e2248d28331461c809f2971" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 42881 } } peers { permanent_uuid: "ff059b5728f94491b9e2d7c8a97c641e" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38205 } } peers { permanent_uuid: "391bb2553f8a44ee837bcd431341b7e5" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45137 } }
I20260619 14:32:10.619917 4371 leader_election.cc:288] T 09aa77aa308a4e94aacaca359f9ba36d P 391bb2553f8a44ee837bcd431341b7e5 [CANDIDATE]: Term 1 election: Requested vote from peers 5bb5a7292e2248d28331461c809f2971 (127.2.2.1:42881), ff059b5728f94491b9e2d7c8a97c641e (127.2.2.2:38205)
I20260619 14:32:10.620357 3952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09aa77aa308a4e94aacaca359f9ba36d" candidate_uuid: "391bb2553f8a44ee837bcd431341b7e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bb5a7292e2248d28331461c809f2971"
I20260619 14:32:10.620357 4093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09aa77aa308a4e94aacaca359f9ba36d" candidate_uuid: "391bb2553f8a44ee837bcd431341b7e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff059b5728f94491b9e2d7c8a97c641e"
I20260619 14:32:10.620498 3952 raft_consensus.cc:3037] T 09aa77aa308a4e94aacaca359f9ba36d P 5bb5a7292e2248d28331461c809f2971 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:10.620515 4093 raft_consensus.cc:3037] T 09aa77aa308a4e94aacaca359f9ba36d P ff059b5728f94491b9e2d7c8a97c641e [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:10.621470 4093 raft_consensus.cc:2454] T 09aa77aa308a4e94aacaca359f9ba36d P ff059b5728f94491b9e2d7c8a97c641e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 391bb2553f8a44ee837bcd431341b7e5 in term 1.
I20260619 14:32:10.621589 3952 raft_consensus.cc:2454] T 09aa77aa308a4e94aacaca359f9ba36d P 5bb5a7292e2248d28331461c809f2971 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 391bb2553f8a44ee837bcd431341b7e5 in term 1.
I20260619 14:32:10.621770 4166 leader_election.cc:302] T 09aa77aa308a4e94aacaca359f9ba36d P 391bb2553f8a44ee837bcd431341b7e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 391bb2553f8a44ee837bcd431341b7e5, ff059b5728f94491b9e2d7c8a97c641e; no voters:
I20260619 14:32:10.621989 4371 raft_consensus.cc:2781] T 09aa77aa308a4e94aacaca359f9ba36d P 391bb2553f8a44ee837bcd431341b7e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:32:10.622083 4371 raft_consensus.cc:686] T 09aa77aa308a4e94aacaca359f9ba36d P 391bb2553f8a44ee837bcd431341b7e5 [term 1 LEADER]: Becoming Leader. State: Replica: 391bb2553f8a44ee837bcd431341b7e5, State: Running, Role: LEADER
I20260619 14:32:10.622246 4371 consensus_queue.cc:237] T 09aa77aa308a4e94aacaca359f9ba36d P 391bb2553f8a44ee837bcd431341b7e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb5a7292e2248d28331461c809f2971" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 42881 } } peers { permanent_uuid: "ff059b5728f94491b9e2d7c8a97c641e" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38205 } } peers { permanent_uuid: "391bb2553f8a44ee837bcd431341b7e5" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45137 } }
I20260619 14:32:10.624001 3795 catalog_manager.cc:5495] T 09aa77aa308a4e94aacaca359f9ba36d P 391bb2553f8a44ee837bcd431341b7e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 391bb2553f8a44ee837bcd431341b7e5 (127.2.2.3). New cstate: current_term: 1 leader_uuid: "391bb2553f8a44ee837bcd431341b7e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bb5a7292e2248d28331461c809f2971" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 42881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff059b5728f94491b9e2d7c8a97c641e" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "391bb2553f8a44ee837bcd431341b7e5" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45137 } health_report { overall_health: HEALTHY } } }
I20260619 14:32:10.644881 4421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:10.641058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52710 (local address 127.2.2.3:45137)
0619 14:32:10.641380 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:10.641387 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:10.641398 (+ 11us) server_negotiation.cc:407] Connection header received
0619 14:32:10.641436 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:10.641438 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:10.641478 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:10.641548 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:32:10.641845 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:10.642424 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:10.643712 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:10.643940 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:10.644021 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:10.644362 (+ 341us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:32:10.644442 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:32:10.644512 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:10.644694 (+ 182us) server_negotiation.cc:299] Negotiation successful
0619 14:32:10.644754 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":168,"threads_started":1}
I20260619 14:32:10.655005 4421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:10.651868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.3:48145 (local address 127.2.2.3:45137)
0619 14:32:10.651976 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:10.651979 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:10.652102 (+ 123us) server_negotiation.cc:407] Connection header received
0619 14:32:10.652250 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:10.652252 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:10.652292 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:10.652383 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:32:10.652666 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:10.653214 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:10.654033 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:10.654282 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:10.654362 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:10.654612 (+ 250us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:32:10.654679 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:32:10.654740 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:10.654838 (+ 98us) server_negotiation.cc:299] Negotiation successful
0619 14:32:10.654895 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":39}
I20260619 14:32:10.655009 4424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:10.651656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.3:45137 (local address 127.2.2.3:48145)
0619 14:32:10.651995 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:10.652013 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:10.652147 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:10.652411 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:10.652414 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:10.652422 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:32:10.652544 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:10.652549 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:10.653314 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:10.653316 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:10.653915 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:10.653922 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:10.654370 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:10.654386 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:10.654464 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:10.654496 (+ 32us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:32:10.654756 (+ 260us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:32:10.654760 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 14:32:10.654809 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 14:32:10.654876 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":19,"thread_start_us":144,"threads_started":1}
I20260619 14:32:10.661144 3952 raft_consensus.cc:1261] T 09aa77aa308a4e94aacaca359f9ba36d P 5bb5a7292e2248d28331461c809f2971 [term 1 FOLLOWER]: Refusing update from remote peer 391bb2553f8a44ee837bcd431341b7e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:32:10.661253 4093 raft_consensus.cc:1261] T 09aa77aa308a4e94aacaca359f9ba36d P ff059b5728f94491b9e2d7c8a97c641e [term 1 FOLLOWER]: Refusing update from remote peer 391bb2553f8a44ee837bcd431341b7e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:32:10.661527 4376 consensus_queue.cc:1035] T 09aa77aa308a4e94aacaca359f9ba36d P 391bb2553f8a44ee837bcd431341b7e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bb5a7292e2248d28331461c809f2971" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 42881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:32:10.662034 4376 consensus_queue.cc:1035] T 09aa77aa308a4e94aacaca359f9ba36d P 391bb2553f8a44ee837bcd431341b7e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff059b5728f94491b9e2d7c8a97c641e" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 14:32:10.677664 4369 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 14:32:12.238893 3795 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:32:12.239044 3795 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:32:12.241506 3795 catalog_manager.cc:5782] T 00000000000000000000000000000000 P de380797462143cc92218c66aee9846e: Sending DeleteTablet for 3 replicas of tablet 09aa77aa308a4e94aacaca359f9ba36d
I20260619 14:32:12.242141 4073 tablet_service.cc:1514] Processing DeleteTablet for tablet 09aa77aa308a4e94aacaca359f9ba36d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:32:12 UTC) from {username='kudu', principal='kudu/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:55986
I20260619 14:32:12.242165 3933 tablet_service.cc:1514] Processing DeleteTablet for tablet 09aa77aa308a4e94aacaca359f9ba36d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:32:12 UTC) from {username='kudu', principal='kudu/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:48050
I20260619 14:32:12.242209 4213 tablet_service.cc:1514] Processing DeleteTablet for tablet 09aa77aa308a4e94aacaca359f9ba36d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:32:12 UTC) from {username='kudu', principal='kudu/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:52688
I20260619 14:32:12.242532 4448 tablet_replica.cc:330] T 09aa77aa308a4e94aacaca359f9ba36d P ff059b5728f94491b9e2d7c8a97c641e: stopping tablet replica
I20260619 14:32:12.242611 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 3868
I20260619 14:32:12.242661 4450 tablet_replica.cc:330] T 09aa77aa308a4e94aacaca359f9ba36d P 391bb2553f8a44ee837bcd431341b7e5: stopping tablet replica
I20260619 14:32:12.243188 4448 raft_consensus.cc:2229] T 09aa77aa308a4e94aacaca359f9ba36d P ff059b5728f94491b9e2d7c8a97c641e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:32:12.243301 4448 raft_consensus.cc:2258] T 09aa77aa308a4e94aacaca359f9ba36d P ff059b5728f94491b9e2d7c8a97c641e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:32:12.243911 4448 ts_tablet_manager.cc:1905] T 09aa77aa308a4e94aacaca359f9ba36d P ff059b5728f94491b9e2d7c8a97c641e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:32:12.244421 4450 raft_consensus.cc:2229] T 09aa77aa308a4e94aacaca359f9ba36d P 391bb2553f8a44ee837bcd431341b7e5 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:32:12.244745 4450 raft_consensus.cc:2258] T 09aa77aa308a4e94aacaca359f9ba36d P 391bb2553f8a44ee837bcd431341b7e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:32:12.245395 4450 ts_tablet_manager.cc:1905] T 09aa77aa308a4e94aacaca359f9ba36d P 391bb2553f8a44ee837bcd431341b7e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:32:12.246668 4448 ts_tablet_manager.cc:1918] T 09aa77aa308a4e94aacaca359f9ba36d P ff059b5728f94491b9e2d7c8a97c641e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 14:32:12.246760 4448 log.cc:1196] T 09aa77aa308a4e94aacaca359f9ba36d P ff059b5728f94491b9e2d7c8a97c641e: Deleting WAL directory at /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/ts-1/wal/wals/09aa77aa308a4e94aacaca359f9ba36d
I20260619 14:32:12.247081 4448 ts_tablet_manager.cc:1939] T 09aa77aa308a4e94aacaca359f9ba36d P ff059b5728f94491b9e2d7c8a97c641e: Deleting consensus metadata
I20260619 14:32:12.247805 3782 catalog_manager.cc:4841] TS ff059b5728f94491b9e2d7c8a97c641e (127.2.2.2:38205): tablet 09aa77aa308a4e94aacaca359f9ba36d (table test-table [id=ab58e486446b4ce583f81d82a92d2d9e]) successfully deleted
I20260619 14:32:12.248072 4450 ts_tablet_manager.cc:1918] T 09aa77aa308a4e94aacaca359f9ba36d P 391bb2553f8a44ee837bcd431341b7e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 14:32:12.248144 4450 log.cc:1196] T 09aa77aa308a4e94aacaca359f9ba36d P 391bb2553f8a44ee837bcd431341b7e5: Deleting WAL directory at /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879514993456-2056-0/minicluster-data/ts-2/wal/wals/09aa77aa308a4e94aacaca359f9ba36d
I20260619 14:32:12.248427 4450 ts_tablet_manager.cc:1939] T 09aa77aa308a4e94aacaca359f9ba36d P 391bb2553f8a44ee837bcd431341b7e5: Deleting consensus metadata
I20260619 14:32:12.249050 3783 catalog_manager.cc:4841] TS 391bb2553f8a44ee837bcd431341b7e5 (127.2.2.3:45137): tablet 09aa77aa308a4e94aacaca359f9ba36d (table test-table [id=ab58e486446b4ce583f81d82a92d2d9e]) successfully deleted
W20260619 14:32:12.250458 3781 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.2.1:42881 (error 108)
I20260619 14:32:12.250728 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 4008
I20260619 14:32:12.252521 4452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:12.252045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.1:42881 (local address 127.0.0.1:48064)
0619 14:32:12.252315 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:12.252401 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.1:42881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":85,"threads_started":1}
W20260619 14:32:12.252727 3781 catalog_manager.cc:4584] TS 5bb5a7292e2248d28331461c809f2971 (127.2.2.1:42881): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 09aa77aa308a4e94aacaca359f9ba36d: Network error: Client connection negotiation failed: client connection to 127.2.2.1:42881: connect: Connection refused (error 111)
I20260619 14:32:12.257575 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 4148
I20260619 14:32:12.264267 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 3764
2026-06-19T14:32:12Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (9288 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:32:12 dist-test-slave-70rm krb5kdc[4455](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:32:12 dist-test-slave-70rm krb5kdc[4455](info): set up 2 sockets
Jun 19 14:32:12 dist-test-slave-70rm krb5kdc[4455](info): commencing operation
krb5kdc: starting...
W20260619 14:32:13.310096 2056 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Jun 19 14:32:13 dist-test-slave-70rm krb5kdc[4455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:32:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:32:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.2.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab.
Entry for principal kudu/127.2.2.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab.
I20260619 14:32:13.434751 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:36725
--webserver_interface=127.2.2.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:46465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.2.62:36725
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:32:13.550257 4468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:13.550597 4468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:13.550680 4468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:13.554874 4468 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:32:13.554981 4468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:13.555032 4468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:13.555089 4468 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:32:13.555145 4468 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:32:13.560650 4468 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:46465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.2.62:36725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:36725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.2.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:32:13.561811 4468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:13.562896 4468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:32:13.571866 4468 server_base.cc:1034] running on GCE node
W20260619 14:32:13.571869 4476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:13.572082 4474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:13.571952 4473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:13.572654 4468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:13.573814 4468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:13.574967 4468 hybrid_clock.cc:648] HybridClock initialized: now 1781879533574934 us; error 48 us; skew 500 ppm
Jun 19 14:32:13 dist-test-slave-70rm krb5kdc[4455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879533, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:32:13.577772 4468 init.cc:377] Logged in from keytab as kudu/127.2.2.62@KRBTEST.COM (short username kudu)
I20260619 14:32:13.578735 4468 webserver.cc:460] Webserver started at http://127.2.2.62:41601/ using document root <none> and password file <none>
I20260619 14:32:13.579349 4468 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:13.579445 4468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:13.579676 4468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:32:13.582561 4468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/master-0/data/instance:
uuid: "1aead9ce114448eca849d6c428f41563"
format_stamp: "Formatted at 2026-06-19 14:32:13 on dist-test-slave-70rm"
server_key: "bcc2a158d1404cfe695f235da4f3e4df"
server_key_iv: "df83a05da16083a0a7dac969701dc230"
server_key_version: "encryptionkey@0"
I20260619 14:32:13.583554 4468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/master-0/wal/instance:
uuid: "1aead9ce114448eca849d6c428f41563"
format_stamp: "Formatted at 2026-06-19 14:32:13 on dist-test-slave-70rm"
server_key: "bcc2a158d1404cfe695f235da4f3e4df"
server_key_iv: "df83a05da16083a0a7dac969701dc230"
server_key_version: "encryptionkey@0"
I20260619 14:32:13.589401 4468 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 14:32:13.592618 4483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:13.593948 4468 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:32:13.594098 4468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/master-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/master-0/wal
uuid: "1aead9ce114448eca849d6c428f41563"
format_stamp: "Formatted at 2026-06-19 14:32:13 on dist-test-slave-70rm"
server_key: "bcc2a158d1404cfe695f235da4f3e4df"
server_key_iv: "df83a05da16083a0a7dac969701dc230"
server_key_version: "encryptionkey@0"
I20260619 14:32:13.594252 4468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:13.621731 4468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:13.625044 4468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:32:13.625267 4468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:13.632624 4468 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.62:36725
I20260619 14:32:13.632642 4535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.62:36725 every 8 connection(s)
I20260619 14:32:13.633653 4468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
I20260619 14:32:13.636721 4536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:13.642038 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 4468
I20260619 14:32:13.642202 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/master-0/wal/instance
I20260619 14:32:13.642458 2056 external_mini_cluster.cc:1428] Setting key 96e88b72fb6a66d4437509778ed9cef5
I20260619 14:32:13.642644 4536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1aead9ce114448eca849d6c428f41563: Bootstrap starting.
I20260619 14:32:13.645219 4536 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1aead9ce114448eca849d6c428f41563: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:13.646111 4536 log.cc:824] T 00000000000000000000000000000000 P 1aead9ce114448eca849d6c428f41563: Log is configured to *not* fsync() on all Append() calls
I20260619 14:32:13.648335 4536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1aead9ce114448eca849d6c428f41563: No bootstrap required, opened a new log
Jun 19 14:32:13 dist-test-slave-70rm krb5kdc[4455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:32:13.653079 4536 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1aead9ce114448eca849d6c428f41563 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aead9ce114448eca849d6c428f41563" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 36725 } }
I20260619 14:32:13.653412 4536 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1aead9ce114448eca849d6c428f41563 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:13.653535 4536 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1aead9ce114448eca849d6c428f41563 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1aead9ce114448eca849d6c428f41563, State: Initialized, Role: FOLLOWER
I20260619 14:32:13.654110 4536 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1aead9ce114448eca849d6c428f41563 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aead9ce114448eca849d6c428f41563" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 36725 } }
I20260619 14:32:13.654295 4536 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1aead9ce114448eca849d6c428f41563 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:32:13.654394 4536 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1aead9ce114448eca849d6c428f41563 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:32:13.654492 4536 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1aead9ce114448eca849d6c428f41563 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:13.655483 4536 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1aead9ce114448eca849d6c428f41563 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aead9ce114448eca849d6c428f41563" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 36725 } }
I20260619 14:32:13.655864 4536 leader_election.cc:302] T 00000000000000000000000000000000 P 1aead9ce114448eca849d6c428f41563 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1aead9ce114448eca849d6c428f41563; no voters:
I20260619 14:32:13.656226 4536 leader_election.cc:288] T 00000000000000000000000000000000 P 1aead9ce114448eca849d6c428f41563 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:32:13.656327 4541 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1aead9ce114448eca849d6c428f41563 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:32:13.656503 4541 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1aead9ce114448eca849d6c428f41563 [term 1 LEADER]: Becoming Leader. State: Replica: 1aead9ce114448eca849d6c428f41563, State: Running, Role: LEADER
I20260619 14:32:13.656881 4541 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1aead9ce114448eca849d6c428f41563 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aead9ce114448eca849d6c428f41563" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 36725 } }
I20260619 14:32:13.657639 4536 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1aead9ce114448eca849d6c428f41563 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:32:13.658608 4542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1aead9ce114448eca849d6c428f41563 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1aead9ce114448eca849d6c428f41563" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aead9ce114448eca849d6c428f41563" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 36725 } } }
I20260619 14:32:13.658758 4542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1aead9ce114448eca849d6c428f41563 [sys.catalog]: This master's current role is: LEADER
I20260619 14:32:13.658993 4542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1aead9ce114448eca849d6c428f41563 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1aead9ce114448eca849d6c428f41563. Latest consensus state: current_term: 1 leader_uuid: "1aead9ce114448eca849d6c428f41563" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aead9ce114448eca849d6c428f41563" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 36725 } } }
I20260619 14:32:13.659008 4539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:13.643936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58732 (local address 127.2.2.62:36725)
0619 14:32:13.644376 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:13.644392 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:13.644430 (+ 38us) server_negotiation.cc:407] Connection header received
0619 14:32:13.645081 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:13.645099 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:13.645518 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:13.645820 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:13.646926 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:13.647967 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:13.649137 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:13.649718 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:13.649860 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:13.652851 (+ 2991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:13.652857 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:13.652873 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:13.652902 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:13.655079 (+ 2177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:13.655578 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:13.655584 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:13.655590 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:13.655747 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:13.656073 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:13.656076 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:13.656078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:13.656661 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:13.657465 (+ 804us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:13.657596 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 14:32:13.657927 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":145,"threads_started":1}
I20260619 14:32:13.659140 4542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1aead9ce114448eca849d6c428f41563 [sys.catalog]: This master's current role is: LEADER
W20260619 14:32:13.663174 4556 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1aead9ce114448eca849d6c428f41563: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:32:13.663278 4556 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:32:13.663386 4550 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:32:13.664208 4550 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:32:13.670346 4550 catalog_manager.cc:1347] Generated new cluster ID: 6d325dff6185428f91cd67f0df435e74
I20260619 14:32:13.670539 4550 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:32:13.683885 4550 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:32:13.685287 4550 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:32:13.692703 4550 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1aead9ce114448eca849d6c428f41563: Generated new TSK 0
I20260619 14:32:13.693595 4550 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.2.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab.
Entry for principal kudu/127.2.2.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab.
I20260619 14:32:13.720971 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.1:0
--local_ip_for_outbound_sockets=127.2.2.1
--webserver_interface=127.2.2.1
--webserver_port=0
--tserver_master_addrs=127.2.2.62:36725
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab
--principal=kudu/127.2.2.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:46465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:32:13.836241 4562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:13.836548 4562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:13.836593 4562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:13.840561 4562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:13.840620 4562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:13.840679 4562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.1
I20260619 14:32:13.845927 4562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:46465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab
--principal=kudu/127.2.2.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.2.1
--webserver_port=0
--tserver_master_addrs=127.2.2.62:36725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.1
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:32:13.847151 4562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:13.848127 4562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:32:13.857975 4567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:13.857986 4568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:13.857968 4570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:13.858067 4562 server_base.cc:1034] running on GCE node
I20260619 14:32:13.858701 4562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:13.859503 4562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:13.860669 4562 hybrid_clock.cc:648] HybridClock initialized: now 1781879533860607 us; error 77 us; skew 500 ppm
Jun 19 14:32:13 dist-test-slave-70rm krb5kdc[4455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879533, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:32:13.863618 4562 init.cc:377] Logged in from keytab as kudu/127.2.2.1@KRBTEST.COM (short username kudu)
I20260619 14:32:13.864530 4562 webserver.cc:460] Webserver started at http://127.2.2.1:45781/ using document root <none> and password file <none>
I20260619 14:32:13.865111 4562 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:13.865198 4562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:13.865422 4562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:32:13.867647 4562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-0/data/instance:
uuid: "41bae6e0a689405987870323041ab45a"
format_stamp: "Formatted at 2026-06-19 14:32:13 on dist-test-slave-70rm"
server_key: "c216ebc9469de50d43b96fadcc6f613c"
server_key_iv: "20fcd7d48cd91d182734421dd58d5bb4"
server_key_version: "encryptionkey@0"
I20260619 14:32:13.868248 4562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-0/wal/instance:
uuid: "41bae6e0a689405987870323041ab45a"
format_stamp: "Formatted at 2026-06-19 14:32:13 on dist-test-slave-70rm"
server_key: "c216ebc9469de50d43b96fadcc6f613c"
server_key_iv: "20fcd7d48cd91d182734421dd58d5bb4"
server_key_version: "encryptionkey@0"
I20260619 14:32:13.872323 4562 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:32:13.875051 4577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:13.876374 4562 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 14:32:13.876477 4562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-0/wal
uuid: "41bae6e0a689405987870323041ab45a"
format_stamp: "Formatted at 2026-06-19 14:32:13 on dist-test-slave-70rm"
server_key: "c216ebc9469de50d43b96fadcc6f613c"
server_key_iv: "20fcd7d48cd91d182734421dd58d5bb4"
server_key_version: "encryptionkey@0"
I20260619 14:32:13.876634 4562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:13.889277 4562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:13.892581 4562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:32:13.892809 4562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:13.893435 4562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:32:13.894382 4562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:32:13.894433 4562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:13.894483 4562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:32:13.894538 4562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:13.903932 4562 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.1:42693
I20260619 14:32:13.903956 4690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.1:42693 every 8 connection(s)
I20260619 14:32:13.904872 4562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
I20260619 14:32:13.907380 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 4562
I20260619 14:32:13.907465 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-0/wal/instance
I20260619 14:32:13.907788 2056 external_mini_cluster.cc:1428] Setting key e83cc1e36cb7cf2769934587e6454b16
Jun 19 14:32:13 dist-test-slave-70rm krb5kdc[4455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879533, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.1@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.2.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.2@KRBTEST.COM" created.
I20260619 14:32:13.919612 4539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:13.906793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:53315 (local address 127.2.2.62:36725)
0619 14:32:13.906964 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:13.906969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:13.907999 (+ 1030us) server_negotiation.cc:407] Connection header received
0619 14:32:13.909664 (+ 1665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:13.909667 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:13.909711 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:13.909778 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:13.911387 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:13.911921 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:13.913127 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:13.913366 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:13.913474 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:13.915939 (+ 2465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:13.915942 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:13.915944 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:13.915967 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:13.917440 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:13.918035 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:13.918040 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:13.918042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:13.918155 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:13.918504 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:13.918508 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:13.918510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:13.918705 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:13.918802 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:13.919315 (+ 513us) server_negotiation.cc:299] Negotiation successful
0619 14:32:13.919449 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 14:32:13.920462 4693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:13.907262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:36725 (local address 127.2.2.1:53315)
0619 14:32:13.907804 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:13.907872 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:13.909434 (+ 1562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:13.909962 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:13.909975 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:13.910569 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:13.911244 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:13.911260 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:13.912046 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:13.912054 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:13.913010 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:13.913019 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:13.913460 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:13.913517 (+ 57us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:13.913686 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:13.914369 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:13.914385 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:13.915767 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:13.917569 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:13.917575 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:13.917592 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:13.917936 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:13.918256 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:13.918259 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:13.918261 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:13.918405 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:13.918812 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:13.918820 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:13.919077 (+ 257us) client_negotiation.cc:769] Sending connection context
0619 14:32:13.919277 (+ 200us) client_negotiation.cc:240] Negotiation successful
0619 14:32:13.919609 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":148,"threads_started":1}
I20260619 14:32:13.922343 4691 heartbeater.cc:347] Connected to a master server at 127.2.2.62:36725
I20260619 14:32:13.922595 4691 heartbeater.cc:464] Registering TS with master...
I20260619 14:32:13.923092 4691 heartbeater.cc:511] Master 127.2.2.62:36725 requested a full tablet report, sending...
I20260619 14:32:13.924865 4500 ts_manager.cc:194] Registered new tserver with Master: 41bae6e0a689405987870323041ab45a (127.2.2.1:42693)
I20260619 14:32:13.926595 4500 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.2.1@KRBTEST.COM'} at 127.2.2.1:53315
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab.
Entry for principal kudu/127.2.2.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab.
I20260619 14:32:13.936575 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.2:0
--local_ip_for_outbound_sockets=127.2.2.2
--webserver_interface=127.2.2.2
--webserver_port=0
--tserver_master_addrs=127.2.2.62:36725
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab
--principal=kudu/127.2.2.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:46465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:32:14.048205 4696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:14.048494 4696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:14.048547 4696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:14.052704 4696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:14.052770 4696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:14.052830 4696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.2
I20260619 14:32:14.058637 4696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:46465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab
--principal=kudu/127.2.2.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.2.2
--webserver_port=0
--tserver_master_addrs=127.2.2.62:36725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.2
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:32:14.059762 4696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:14.060806 4696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:32:14.070670 4701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:14.070693 4704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:14.070672 4702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:14.070730 4696 server_base.cc:1034] running on GCE node
I20260619 14:32:14.071295 4696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:14.071939 4696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:14.073155 4696 hybrid_clock.cc:648] HybridClock initialized: now 1781879534073137 us; error 31 us; skew 500 ppm
Jun 19 14:32:14 dist-test-slave-70rm krb5kdc[4455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879534, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:32:14.076318 4696 init.cc:377] Logged in from keytab as kudu/127.2.2.2@KRBTEST.COM (short username kudu)
I20260619 14:32:14.077329 4696 webserver.cc:460] Webserver started at http://127.2.2.2:37201/ using document root <none> and password file <none>
I20260619 14:32:14.077952 4696 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:14.078069 4696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:14.078352 4696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:32:14.080672 4696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-1/data/instance:
uuid: "5558246893df44139425fb1df5ec6349"
format_stamp: "Formatted at 2026-06-19 14:32:14 on dist-test-slave-70rm"
server_key: "d5e4ffd2f72c71f4d2925cdecb2f498b"
server_key_iv: "63e49b3c61c3db04f85bcb275fe8c303"
server_key_version: "encryptionkey@0"
I20260619 14:32:14.081346 4696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-1/wal/instance:
uuid: "5558246893df44139425fb1df5ec6349"
format_stamp: "Formatted at 2026-06-19 14:32:14 on dist-test-slave-70rm"
server_key: "d5e4ffd2f72c71f4d2925cdecb2f498b"
server_key_iv: "63e49b3c61c3db04f85bcb275fe8c303"
server_key_version: "encryptionkey@0"
I20260619 14:32:14.085589 4696 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 14:32:14.088750 4711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:14.090082 4696 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:32:14.090296 4696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-1/wal
uuid: "5558246893df44139425fb1df5ec6349"
format_stamp: "Formatted at 2026-06-19 14:32:14 on dist-test-slave-70rm"
server_key: "d5e4ffd2f72c71f4d2925cdecb2f498b"
server_key_iv: "63e49b3c61c3db04f85bcb275fe8c303"
server_key_version: "encryptionkey@0"
I20260619 14:32:14.090435 4696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:14.102522 4696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:14.105970 4696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:32:14.106254 4696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:14.106897 4696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:32:14.107694 4696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:32:14.107740 4696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:14.107780 4696 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:32:14.107795 4696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:14.117017 4696 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.2:34589
I20260619 14:32:14.117036 4824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.2:34589 every 8 connection(s)
I20260619 14:32:14.117959 4696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
I20260619 14:32:14.123063 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 4696
I20260619 14:32:14.123159 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-1/wal/instance
I20260619 14:32:14.123456 2056 external_mini_cluster.cc:1428] Setting key ffced5f8dd065bdef8b876f4e10563a1
Jun 19 14:32:14 dist-test-slave-70rm krb5kdc[4455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879534, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.2@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:32:14.133317 4539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:14.119850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:48603 (local address 127.2.2.62:36725)
0619 14:32:14.120008 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:14.120012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:14.120931 (+ 919us) server_negotiation.cc:407] Connection header received
0619 14:32:14.122250 (+ 1319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:14.122253 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:14.122301 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:14.122377 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:14.123698 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:14.124281 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:14.125639 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:14.126034 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:14.126175 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:14.129059 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:14.129065 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:14.129070 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:14.129104 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:14.131019 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:14.131563 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:14.131568 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:14.131570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:14.131667 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:14.132000 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:14.132006 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:14.132009 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:14.132267 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:14.132370 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:14.132916 (+ 546us) server_negotiation.cc:299] Negotiation successful
0619 14:32:14.133067 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
WARNING: no policy specified for kudu/127.2.2.3@KRBTEST.COM; defaulting to no policy
I20260619 14:32:14.134200 4827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:14.120303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:36725 (local address 127.2.2.2:48603)
0619 14:32:14.120766 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:14.120811 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:14.122016 (+ 1205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:14.122542 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:14.122549 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:14.122937 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:14.123541 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:14.123563 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:14.124401 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:14.124404 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:14.125510 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:14.125519 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:14.126213 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:14.126258 (+ 45us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:14.126403 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:14.127080 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:14.127104 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:14.128821 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:14.131167 (+ 2346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:14.131172 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:14.131186 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:14.131456 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:14.131767 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:14.131769 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:14.131770 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:14.131904 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:14.132374 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:14.132379 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:14.132657 (+ 278us) client_negotiation.cc:769] Sending connection context
0619 14:32:14.132820 (+ 163us) client_negotiation.cc:240] Negotiation successful
0619 14:32:14.133091 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":125,"threads_started":1}
I20260619 14:32:14.135526 4825 heartbeater.cc:347] Connected to a master server at 127.2.2.62:36725
I20260619 14:32:14.135813 4825 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.3@KRBTEST.COM" created.
I20260619 14:32:14.136466 4825 heartbeater.cc:511] Master 127.2.2.62:36725 requested a full tablet report, sending...
I20260619 14:32:14.137595 4500 ts_manager.cc:194] Registered new tserver with Master: 5558246893df44139425fb1df5ec6349 (127.2.2.2:34589)
I20260619 14:32:14.138242 4500 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.2.2@KRBTEST.COM'} at 127.2.2.2:48603
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab.
Entry for principal kudu/127.2.2.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab.
I20260619 14:32:14.153327 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.3:0
--local_ip_for_outbound_sockets=127.2.2.3
--webserver_interface=127.2.2.3
--webserver_port=0
--tserver_master_addrs=127.2.2.62:36725
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab
--principal=kudu/127.2.2.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:46465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:32:14.265935 4830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:14.266261 4830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:14.266345 4830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:14.270398 4830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:14.270494 4830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:14.270597 4830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.3
I20260619 14:32:14.275825 4830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:46465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab
--principal=kudu/127.2.2.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.2.3
--webserver_port=0
--tserver_master_addrs=127.2.2.62:36725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.3
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:32:14.276917 4830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:14.278038 4830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:32:14.288234 4838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:14.288311 4836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:14.288313 4830 server_base.cc:1034] running on GCE node
W20260619 14:32:14.288693 4835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:14.289247 4830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:14.289872 4830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:14.291028 4830 hybrid_clock.cc:648] HybridClock initialized: now 1781879534290997 us; error 48 us; skew 500 ppm
Jun 19 14:32:14 dist-test-slave-70rm krb5kdc[4455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879534, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:32:14.293840 4830 init.cc:377] Logged in from keytab as kudu/127.2.2.3@KRBTEST.COM (short username kudu)
I20260619 14:32:14.294847 4830 webserver.cc:460] Webserver started at http://127.2.2.3:45093/ using document root <none> and password file <none>
I20260619 14:32:14.295399 4830 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:14.295456 4830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:14.295616 4830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:32:14.297736 4830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-2/data/instance:
uuid: "c00e097e8d554970957d6ee34234ac50"
format_stamp: "Formatted at 2026-06-19 14:32:14 on dist-test-slave-70rm"
server_key: "0f6dc1855ea2eedc823cbc55e34d3817"
server_key_iv: "ae2bc5da9724f67e8f6956ed45d0e306"
server_key_version: "encryptionkey@0"
I20260619 14:32:14.298388 4830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-2/wal/instance:
uuid: "c00e097e8d554970957d6ee34234ac50"
format_stamp: "Formatted at 2026-06-19 14:32:14 on dist-test-slave-70rm"
server_key: "0f6dc1855ea2eedc823cbc55e34d3817"
server_key_iv: "ae2bc5da9724f67e8f6956ed45d0e306"
server_key_version: "encryptionkey@0"
I20260619 14:32:14.302390 4830 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:32:14.304834 4845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:14.306238 4830 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:32:14.306365 4830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-2/wal
uuid: "c00e097e8d554970957d6ee34234ac50"
format_stamp: "Formatted at 2026-06-19 14:32:14 on dist-test-slave-70rm"
server_key: "0f6dc1855ea2eedc823cbc55e34d3817"
server_key_iv: "ae2bc5da9724f67e8f6956ed45d0e306"
server_key_version: "encryptionkey@0"
I20260619 14:32:14.306481 4830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:14.324401 4830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:14.327840 4830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:32:14.328056 4830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:14.328670 4830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:32:14.329582 4830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:32:14.329649 4830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:14.329721 4830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:32:14.329772 4830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:14.338953 4830 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.3:38389
I20260619 14:32:14.338974 4958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.3:38389 every 8 connection(s)
I20260619 14:32:14.339990 4830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
Jun 19 14:32:14 dist-test-slave-70rm krb5kdc[4455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879534, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.3@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:32:14.349910 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 4830
I20260619 14:32:14.350057 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-2/wal/instance
I20260619 14:32:14.350354 2056 external_mini_cluster.cc:1428] Setting key 2547ebaf7488c4f6a816967fc967123d
I20260619 14:32:14.354429 4539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:14.341984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.3:39397 (local address 127.2.2.62:36725)
0619 14:32:14.342189 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:14.342194 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:14.343065 (+ 871us) server_negotiation.cc:407] Connection header received
0619 14:32:14.344413 (+ 1348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:14.344416 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:14.344465 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:14.344561 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:14.345771 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:14.346351 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:14.347482 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:14.347719 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:14.347822 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:14.350626 (+ 2804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:14.350629 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:14.350631 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:14.350681 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:14.352456 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:14.352998 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:14.353001 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:14.353002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:14.353090 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:14.353417 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:14.353420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:14.353421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:14.353612 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:14.353691 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:14.354192 (+ 501us) server_negotiation.cc:299] Negotiation successful
0619 14:32:14.354311 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 14:32:14.355273 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:14.342446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:36725 (local address 127.2.2.3:39397)
0619 14:32:14.342907 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:14.342950 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:14.344173 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:14.344728 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:14.344735 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:14.345099 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:14.345624 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:14.345636 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:14.346463 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:14.346465 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:14.347359 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:14.347368 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:14.347819 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:14.347866 (+ 47us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:14.348005 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:14.348668 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:14.348684 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:14.350437 (+ 1753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:14.352593 (+ 2156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:14.352600 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:14.352616 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:14.352894 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:14.353177 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:14.353179 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:14.353180 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:14.353322 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:14.353698 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:14.353703 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:14.353930 (+ 227us) client_negotiation.cc:769] Sending connection context
0619 14:32:14.354085 (+ 155us) client_negotiation.cc:240] Negotiation successful
0619 14:32:14.354408 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":115,"threads_started":1}
I20260619 14:32:14.356360 4959 heartbeater.cc:347] Connected to a master server at 127.2.2.62:36725
I20260619 14:32:14.356602 4959 heartbeater.cc:464] Registering TS with master...
I20260619 14:32:14.357101 4959 heartbeater.cc:511] Master 127.2.2.62:36725 requested a full tablet report, sending...
I20260619 14:32:14.358114 4500 ts_manager.cc:194] Registered new tserver with Master: c00e097e8d554970957d6ee34234ac50 (127.2.2.3:38389)
I20260619 14:32:14.358716 4500 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.2.3@KRBTEST.COM'} at 127.2.2.3:39397
I20260619 14:32:14.364513 2056 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 14:32:14.377969 4539 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:32:14.376258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58734 (local address 127.2.2.62:36725)
0619 14:32:14.376496 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:14.376501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:14.376637 (+ 136us) server_negotiation.cc:407] Connection header received
0619 14:32:14.376832 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:14.376836 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:14.376891 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:14.376980 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:14.377804 (+ 824us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58734: BlockingRecv error: recv got EOF from 127.0.0.1:58734 (error 108)
Metrics: {"server-negotiator.queue_time_us":131}
I20260619 14:32:14.379153 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 4562
I20260619 14:32:14.384846 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 4696
I20260619 14:32:14.390251 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 4830
I20260619 14:32:14.395562 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 4468
2026-06-19T14:32:14Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2129 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781879514993456-2056-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:32:14 dist-test-slave-70rm krb5kdc[4971](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:32:14 dist-test-slave-70rm krb5kdc[4971](info): set up 2 sockets
Jun 19 14:32:14 dist-test-slave-70rm krb5kdc[4971](info): commencing operation
krb5kdc: starting...
W20260619 14:32:15.450304 2056 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Jun 19 14:32:15 dist-test-slave-70rm krb5kdc[4971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:32:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:32:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.2.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab.
Entry for principal kudu/127.2.2.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab.
I20260619 14:32:15.577952 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781879514993456-2056-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781879514993456-2056-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781879514993456-2056-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:36177
--webserver_interface=127.2.2.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:45109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.2.62:36177
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:32:15.700901 4984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:15.701218 4984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:15.701261 4984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:15.705447 4984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:32:15.705515 4984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:15.705534 4984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:15.705551 4984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:32:15.705567 4984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:32:15.710950 4984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:45109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781879514993456-2056-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781879514993456-2056-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.2.62:36177
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:36177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.2.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781879514993456-2056-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:32:15.712056 4984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:15.713071 4984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:32:15.721690 4989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:15.721699 4992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:15.721819 4984 server_base.cc:1034] running on GCE node
W20260619 14:32:15.721700 4990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:15.722645 4984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:15.723816 4984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:15.725060 4984 hybrid_clock.cc:648] HybridClock initialized: now 1781879535725039 us; error 59 us; skew 500 ppm
Jun 19 14:32:15 dist-test-slave-70rm krb5kdc[4971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879535, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:32:15.728082 4984 init.cc:377] Logged in from keytab as kudu/127.2.2.62@KRBTEST.COM (short username kudu)
I20260619 14:32:15.729063 4984 webserver.cc:460] Webserver started at http://127.2.2.62:39911/ using document root <none> and password file <none>
I20260619 14:32:15.729602 4984 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:15.729665 4984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:15.729836 4984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:32:15.732017 4984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781879514993456-2056-0/minicluster-data/master-0/data/instance:
uuid: "f6b9dad29a38494d807dd7a11278b266"
format_stamp: "Formatted at 2026-06-19 14:32:15 on dist-test-slave-70rm"
server_key: "dad2892c1687c40e6f075b0ccc6436c2"
server_key_iv: "8f86646cedfa14648b41f27136113853"
server_key_version: "encryptionkey@0"
I20260619 14:32:15.732535 4984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781879514993456-2056-0/minicluster-data/master-0/wal/instance:
uuid: "f6b9dad29a38494d807dd7a11278b266"
format_stamp: "Formatted at 2026-06-19 14:32:15 on dist-test-slave-70rm"
server_key: "dad2892c1687c40e6f075b0ccc6436c2"
server_key_iv: "8f86646cedfa14648b41f27136113853"
server_key_version: "encryptionkey@0"
I20260619 14:32:15.736449 4984 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:32:15.738919 4999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:15.740325 4984 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:32:15.740455 4984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781879514993456-2056-0/minicluster-data/master-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781879514993456-2056-0/minicluster-data/master-0/wal
uuid: "f6b9dad29a38494d807dd7a11278b266"
format_stamp: "Formatted at 2026-06-19 14:32:15 on dist-test-slave-70rm"
server_key: "dad2892c1687c40e6f075b0ccc6436c2"
server_key_iv: "8f86646cedfa14648b41f27136113853"
server_key_version: "encryptionkey@0"
I20260619 14:32:15.740562 4984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781879514993456-2056-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781879514993456-2056-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781879514993456-2056-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:15.750432 4984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:15.753628 4984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:32:15.753829 4984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:15.761049 4984 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.62:36177
I20260619 14:32:15.761335 5051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.62:36177 every 8 connection(s)
I20260619 14:32:15.762100 4984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
I20260619 14:32:15.764942 5052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:15.765338 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 4984
I20260619 14:32:15.765460 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781879514993456-2056-0/minicluster-data/master-0/wal/instance
I20260619 14:32:15.765745 2056 external_mini_cluster.cc:1428] Setting key f0f8a3063cadee24452d7126e64e1ce8
I20260619 14:32:15.771768 5052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6b9dad29a38494d807dd7a11278b266: Bootstrap starting.
I20260619 14:32:15.774387 5052 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6b9dad29a38494d807dd7a11278b266: Neither blocks nor log segments found. Creating new log.
Jun 19 14:32:15 dist-test-slave-70rm krb5kdc[4971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:32:15.775378 5052 log.cc:824] T 00000000000000000000000000000000 P f6b9dad29a38494d807dd7a11278b266: Log is configured to *not* fsync() on all Append() calls
I20260619 14:32:15.777812 5052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6b9dad29a38494d807dd7a11278b266: No bootstrap required, opened a new log
I20260619 14:32:15.780503 5055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:15.767226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41176 (local address 127.2.2.62:36177)
0619 14:32:15.767680 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:15.767695 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:15.767731 (+ 36us) server_negotiation.cc:407] Connection header received
0619 14:32:15.768436 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:15.768455 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:15.768809 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:15.769158 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:15.769973 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:15.771052 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:15.771931 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:15.772431 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:15.772616 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:15.775542 (+ 2926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:15.775552 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:15.775587 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:15.775618 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:15.777406 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:15.777949 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:15.777955 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:15.777963 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:15.778104 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:15.778486 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:15.778489 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:15.778490 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:15.778931 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:15.779096 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:15.779402 (+ 306us) server_negotiation.cc:299] Negotiation successful
0619 14:32:15.779747 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":130,"threads_started":1}
I20260619 14:32:15.781898 5052 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6b9dad29a38494d807dd7a11278b266 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6b9dad29a38494d807dd7a11278b266" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 36177 } }
I20260619 14:32:15.782186 5052 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6b9dad29a38494d807dd7a11278b266 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:15.782274 5052 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6b9dad29a38494d807dd7a11278b266 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b9dad29a38494d807dd7a11278b266, State: Initialized, Role: FOLLOWER
I20260619 14:32:15.782742 5052 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6b9dad29a38494d807dd7a11278b266 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6b9dad29a38494d807dd7a11278b266" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 36177 } }
I20260619 14:32:15.782923 5052 raft_consensus.cc:388] T 00000000000000000000000000000000 P f6b9dad29a38494d807dd7a11278b266 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:32:15.782997 5052 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6b9dad29a38494d807dd7a11278b266 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:32:15.783100 5052 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6b9dad29a38494d807dd7a11278b266 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:15.783504 2056 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:32:15.784345 5052 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6b9dad29a38494d807dd7a11278b266 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6b9dad29a38494d807dd7a11278b266" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 36177 } }
I20260619 14:32:15.784758 5052 leader_election.cc:302] T 00000000000000000000000000000000 P f6b9dad29a38494d807dd7a11278b266 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6b9dad29a38494d807dd7a11278b266; no voters:
I20260619 14:32:15.785143 5052 leader_election.cc:288] T 00000000000000000000000000000000 P f6b9dad29a38494d807dd7a11278b266 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:32:15.785277 5057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6b9dad29a38494d807dd7a11278b266 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:32:15.785530 5057 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6b9dad29a38494d807dd7a11278b266 [term 1 LEADER]: Becoming Leader. State: Replica: f6b9dad29a38494d807dd7a11278b266, State: Running, Role: LEADER
I20260619 14:32:15.785918 5057 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6b9dad29a38494d807dd7a11278b266 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6b9dad29a38494d807dd7a11278b266" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 36177 } }
I20260619 14:32:15.786530 5052 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6b9dad29a38494d807dd7a11278b266 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:32:15.787791 5058 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6b9dad29a38494d807dd7a11278b266 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6b9dad29a38494d807dd7a11278b266" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6b9dad29a38494d807dd7a11278b266" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 36177 } } }
I20260619 14:32:15.787919 5058 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6b9dad29a38494d807dd7a11278b266 [sys.catalog]: This master's current role is: LEADER
I20260619 14:32:15.788255 5060 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6b9dad29a38494d807dd7a11278b266 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6b9dad29a38494d807dd7a11278b266. Latest consensus state: current_term: 1 leader_uuid: "f6b9dad29a38494d807dd7a11278b266" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6b9dad29a38494d807dd7a11278b266" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 36177 } } }
I20260619 14:32:15.788347 5060 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6b9dad29a38494d807dd7a11278b266 [sys.catalog]: This master's current role is: LEADER
W20260619 14:32:15.792168 5073 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f6b9dad29a38494d807dd7a11278b266: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:32:15.792260 5073 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:32:15.792351 5067 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:32:15.793061 5067 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:32:15.799011 5067 catalog_manager.cc:1347] Generated new cluster ID: 97fda85e36134a8eba84afd82448ee7a
I20260619 14:32:15.799099 5067 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:32:15.805507 5055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:15.799119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41178 (local address 127.2.2.62:36177)
0619 14:32:15.799243 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:15.799248 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:15.799877 (+ 629us) server_negotiation.cc:407] Connection header received
0619 14:32:15.800017 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:15.800020 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:15.800074 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:15.800171 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:15.800803 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:15.801669 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:15.802889 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:15.803264 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:15.803384 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:15.804332 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:15.804336 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:15.804339 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:32:15.805013 (+ 674us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:15.805053 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:32:15.805074 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:15.805171 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:15.805278 (+ 107us) server_negotiation.cc:299] Negotiation successful
0619 14:32:15.805351 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 14:32:15.808123 5067 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:32:15.809023 5067 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:32:15.814173 5067 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6b9dad29a38494d807dd7a11278b266: Generated new TSK 0
I20260619 14:32:15.814888 5067 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:32:15.819942 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 4984
2026-06-19T14:32:15Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1422 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:32:15 dist-test-slave-70rm krb5kdc[5085](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:32:15 dist-test-slave-70rm krb5kdc[5085](info): set up 2 sockets
Jun 19 14:32:15 dist-test-slave-70rm krb5kdc[5085](info): commencing operation
krb5kdc: starting...
W20260619 14:32:16.858464 2056 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Jun 19 14:32:16 dist-test-slave-70rm krb5kdc[5085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:32:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:32:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.2.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab.
Entry for principal kudu/127.2.2.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab.
I20260619 14:32:16.985100 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:33859
--webserver_interface=127.2.2.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:34685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.2.62:33859
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:32:17.095687 5098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:17.095983 5098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:17.096026 5098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:17.100210 5098 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:32:17.100293 5098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:17.100313 5098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:17.100333 5098 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:32:17.100349 5098 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:32:17.105888 5098 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:34685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.2.62:33859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:33859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.2.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:32:17.107069 5098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:17.108218 5098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:32:17.117189 5098 server_base.cc:1034] running on GCE node
W20260619 14:32:17.117146 5103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:17.117146 5104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:17.117151 5106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:17.117964 5098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:17.119200 5098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:17.120409 5098 hybrid_clock.cc:648] HybridClock initialized: now 1781879537120397 us; error 32 us; skew 500 ppm
Jun 19 14:32:17 dist-test-slave-70rm krb5kdc[5085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879537, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:32:17.123454 5098 init.cc:377] Logged in from keytab as kudu/127.2.2.62@KRBTEST.COM (short username kudu)
I20260619 14:32:17.124457 5098 webserver.cc:460] Webserver started at http://127.2.2.62:45725/ using document root <none> and password file <none>
I20260619 14:32:17.125068 5098 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:17.125164 5098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:17.125396 5098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:32:17.127548 5098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/master-0/data/instance:
uuid: "b31ab3d91e04456eb6782e0278a496ba"
format_stamp: "Formatted at 2026-06-19 14:32:17 on dist-test-slave-70rm"
server_key: "29cd43907c5ca1658880e69217466048"
server_key_iv: "5978af6d9470a273c1d5447255388582"
server_key_version: "encryptionkey@0"
I20260619 14:32:17.128134 5098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/master-0/wal/instance:
uuid: "b31ab3d91e04456eb6782e0278a496ba"
format_stamp: "Formatted at 2026-06-19 14:32:17 on dist-test-slave-70rm"
server_key: "29cd43907c5ca1658880e69217466048"
server_key_iv: "5978af6d9470a273c1d5447255388582"
server_key_version: "encryptionkey@0"
I20260619 14:32:17.132059 5098 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:32:17.134696 5113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:17.135926 5098 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:32:17.136076 5098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/master-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/master-0/wal
uuid: "b31ab3d91e04456eb6782e0278a496ba"
format_stamp: "Formatted at 2026-06-19 14:32:17 on dist-test-slave-70rm"
server_key: "29cd43907c5ca1658880e69217466048"
server_key_iv: "5978af6d9470a273c1d5447255388582"
server_key_version: "encryptionkey@0"
I20260619 14:32:17.136197 5098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:17.165702 5098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:17.169401 5098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:32:17.169641 5098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:17.176968 5098 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.62:33859
I20260619 14:32:17.177273 5165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.62:33859 every 8 connection(s)
I20260619 14:32:17.178057 5098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
I20260619 14:32:17.181015 5166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:17.181945 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 5098
I20260619 14:32:17.182061 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/master-0/wal/instance
I20260619 14:32:17.182386 2056 external_mini_cluster.cc:1428] Setting key 03e769ba56768b4fa2aaccb83d6c4a62
I20260619 14:32:17.187855 5166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b31ab3d91e04456eb6782e0278a496ba: Bootstrap starting.
I20260619 14:32:17.190784 5166 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b31ab3d91e04456eb6782e0278a496ba: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:17.191718 5166 log.cc:824] T 00000000000000000000000000000000 P b31ab3d91e04456eb6782e0278a496ba: Log is configured to *not* fsync() on all Append() calls
Jun 19 14:32:17 dist-test-slave-70rm krb5kdc[5085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:32:17.193800 5166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b31ab3d91e04456eb6782e0278a496ba: No bootstrap required, opened a new log
I20260619 14:32:17.197547 5169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:17.183867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42240 (local address 127.2.2.62:33859)
0619 14:32:17.184315 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:17.184331 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:17.184369 (+ 38us) server_negotiation.cc:407] Connection header received
0619 14:32:17.185006 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:17.185033 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:17.185379 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:17.185677 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:17.186689 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:17.187731 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:17.188910 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:17.189413 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:17.189572 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:17.192635 (+ 3063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:17.192641 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:17.192657 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:17.192683 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:17.194609 (+ 1926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:17.195110 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:17.195115 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:17.195120 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:17.195234 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:17.195530 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:17.195532 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:17.195533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:17.195926 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:17.196080 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:17.196435 (+ 355us) server_negotiation.cc:299] Negotiation successful
0619 14:32:17.196772 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":154,"threads_started":1}
I20260619 14:32:17.198055 5166 raft_consensus.cc:348] T 00000000000000000000000000000000 P b31ab3d91e04456eb6782e0278a496ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b31ab3d91e04456eb6782e0278a496ba" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 33859 } }
I20260619 14:32:17.198365 5166 raft_consensus.cc:374] T 00000000000000000000000000000000 P b31ab3d91e04456eb6782e0278a496ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:17.198460 5166 raft_consensus.cc:729] T 00000000000000000000000000000000 P b31ab3d91e04456eb6782e0278a496ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b31ab3d91e04456eb6782e0278a496ba, State: Initialized, Role: FOLLOWER
I20260619 14:32:17.198904 5166 consensus_queue.cc:260] T 00000000000000000000000000000000 P b31ab3d91e04456eb6782e0278a496ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b31ab3d91e04456eb6782e0278a496ba" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 33859 } }
I20260619 14:32:17.199028 5166 raft_consensus.cc:388] T 00000000000000000000000000000000 P b31ab3d91e04456eb6782e0278a496ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:32:17.199112 5166 raft_consensus.cc:482] T 00000000000000000000000000000000 P b31ab3d91e04456eb6782e0278a496ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:32:17.199231 5166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b31ab3d91e04456eb6782e0278a496ba [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:17.200201 5166 raft_consensus.cc:504] T 00000000000000000000000000000000 P b31ab3d91e04456eb6782e0278a496ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b31ab3d91e04456eb6782e0278a496ba" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 33859 } }
I20260619 14:32:17.200549 5166 leader_election.cc:302] T 00000000000000000000000000000000 P b31ab3d91e04456eb6782e0278a496ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b31ab3d91e04456eb6782e0278a496ba; no voters:
I20260619 14:32:17.200866 5166 leader_election.cc:288] T 00000000000000000000000000000000 P b31ab3d91e04456eb6782e0278a496ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:32:17.201148 5171 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b31ab3d91e04456eb6782e0278a496ba [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:32:17.201390 5171 raft_consensus.cc:686] T 00000000000000000000000000000000 P b31ab3d91e04456eb6782e0278a496ba [term 1 LEADER]: Becoming Leader. State: Replica: b31ab3d91e04456eb6782e0278a496ba, State: Running, Role: LEADER
I20260619 14:32:17.201833 5171 consensus_queue.cc:237] T 00000000000000000000000000000000 P b31ab3d91e04456eb6782e0278a496ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b31ab3d91e04456eb6782e0278a496ba" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 33859 } }
I20260619 14:32:17.202390 5166 sys_catalog.cc:564] T 00000000000000000000000000000000 P b31ab3d91e04456eb6782e0278a496ba [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:32:17.204069 5172 sys_catalog.cc:455] T 00000000000000000000000000000000 P b31ab3d91e04456eb6782e0278a496ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader b31ab3d91e04456eb6782e0278a496ba. Latest consensus state: current_term: 1 leader_uuid: "b31ab3d91e04456eb6782e0278a496ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b31ab3d91e04456eb6782e0278a496ba" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 33859 } } }
I20260619 14:32:17.204073 5173 sys_catalog.cc:455] T 00000000000000000000000000000000 P b31ab3d91e04456eb6782e0278a496ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b31ab3d91e04456eb6782e0278a496ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b31ab3d91e04456eb6782e0278a496ba" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 33859 } } }
I20260619 14:32:17.204234 5173 sys_catalog.cc:458] T 00000000000000000000000000000000 P b31ab3d91e04456eb6782e0278a496ba [sys.catalog]: This master's current role is: LEADER
I20260619 14:32:17.204234 5172 sys_catalog.cc:458] T 00000000000000000000000000000000 P b31ab3d91e04456eb6782e0278a496ba [sys.catalog]: This master's current role is: LEADER
W20260619 14:32:17.207026 5186 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b31ab3d91e04456eb6782e0278a496ba: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:32:17.207099 5186 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:32:17.207187 5185 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:32:17.208030 5185 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:32:17.213755 5185 catalog_manager.cc:1347] Generated new cluster ID: 76cb0ff9196a40e3b08db6f22d9fcb8e
I20260619 14:32:17.213840 5185 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:32:17.229977 5185 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:32:17.230983 5185 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:32:17.236209 5185 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b31ab3d91e04456eb6782e0278a496ba: Generated new TSK 0
I20260619 14:32:17.237044 5185 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.2.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab.
Entry for principal kudu/127.2.2.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab.
I20260619 14:32:17.269510 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.1:0
--local_ip_for_outbound_sockets=127.2.2.1
--webserver_interface=127.2.2.1
--webserver_port=0
--tserver_master_addrs=127.2.2.62:33859
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab
--principal=kudu/127.2.2.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:34685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:32:17.385162 5192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:17.385452 5192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:17.385497 5192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:17.389714 5192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:17.389801 5192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:17.389864 5192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.1
I20260619 14:32:17.395539 5192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:34685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab
--principal=kudu/127.2.2.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.2.1
--webserver_port=0
--tserver_master_addrs=127.2.2.62:33859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.1
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:32:17.396699 5192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:17.397778 5192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:32:17.407873 5200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:17.407984 5198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:17.407908 5197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:17.408202 5192 server_base.cc:1034] running on GCE node
I20260619 14:32:17.408622 5192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:17.409274 5192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:17.410414 5192 hybrid_clock.cc:648] HybridClock initialized: now 1781879537410381 us; error 48 us; skew 500 ppm
Jun 19 14:32:17 dist-test-slave-70rm krb5kdc[5085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879537, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:32:17.418828 5192 init.cc:377] Logged in from keytab as kudu/127.2.2.1@KRBTEST.COM (short username kudu)
I20260619 14:32:17.419746 5192 webserver.cc:460] Webserver started at http://127.2.2.1:42331/ using document root <none> and password file <none>
I20260619 14:32:17.420305 5192 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:17.420370 5192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:17.420552 5192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:32:17.422719 5192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-0/data/instance:
uuid: "814e118005e8441cacedd828aec64f2d"
format_stamp: "Formatted at 2026-06-19 14:32:17 on dist-test-slave-70rm"
server_key: "346cd84c08a7e469653191fd689311d0"
server_key_iv: "97bc65f80eb7bba3b17e172cc47fe748"
server_key_version: "encryptionkey@0"
I20260619 14:32:17.423254 5192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-0/wal/instance:
uuid: "814e118005e8441cacedd828aec64f2d"
format_stamp: "Formatted at 2026-06-19 14:32:17 on dist-test-slave-70rm"
server_key: "346cd84c08a7e469653191fd689311d0"
server_key_iv: "97bc65f80eb7bba3b17e172cc47fe748"
server_key_version: "encryptionkey@0"
I20260619 14:32:17.427165 5192 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 14:32:17.429908 5207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:17.431389 5192 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:32:17.431550 5192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-0/wal
uuid: "814e118005e8441cacedd828aec64f2d"
format_stamp: "Formatted at 2026-06-19 14:32:17 on dist-test-slave-70rm"
server_key: "346cd84c08a7e469653191fd689311d0"
server_key_iv: "97bc65f80eb7bba3b17e172cc47fe748"
server_key_version: "encryptionkey@0"
I20260619 14:32:17.431669 5192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:17.448787 5192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:17.452255 5192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:32:17.452464 5192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:17.453075 5192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:32:17.453917 5192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:32:17.453963 5192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:17.454001 5192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:32:17.454017 5192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:17.463209 5192 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.1:38497
I20260619 14:32:17.463227 5320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.1:38497 every 8 connection(s)
I20260619 14:32:17.464139 5192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
I20260619 14:32:17.466331 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 5192
I20260619 14:32:17.466446 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-0/wal/instance
I20260619 14:32:17.466758 2056 external_mini_cluster.cc:1428] Setting key 1e46f266228dce434f1bbbd742b93bfa
Jun 19 14:32:17 dist-test-slave-70rm krb5kdc[5085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879537, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.1@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.2.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.2@KRBTEST.COM" created.
I20260619 14:32:17.479960 5169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:17.465996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:35727 (local address 127.2.2.62:33859)
0619 14:32:17.466180 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:17.466188 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:17.467315 (+ 1127us) server_negotiation.cc:407] Connection header received
0619 14:32:17.468783 (+ 1468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:17.468787 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:17.468841 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:17.468920 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:17.470114 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:17.470686 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:17.471990 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:17.472421 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:17.472563 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:17.475451 (+ 2888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:17.475458 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:17.475464 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:17.475502 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:17.477046 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:17.477797 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:17.477801 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:17.477802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:17.477879 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:17.478349 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:17.478353 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:17.478355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:17.478662 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:17.478819 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:17.479572 (+ 753us) server_negotiation.cc:299] Negotiation successful
0619 14:32:17.479789 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 14:32:17.481012 5323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:17.466568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:33859 (local address 127.2.2.1:35727)
0619 14:32:17.467104 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:17.467183 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:17.468567 (+ 1384us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:17.469109 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:17.469116 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:17.469466 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:17.469962 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:17.469976 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:17.470837 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:17.470840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:17.471848 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:17.471861 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:17.472560 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:17.472610 (+ 50us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:17.472771 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:17.473450 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:17.473467 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:17.475240 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:17.477186 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:17.477196 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:17.477218 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:17.477677 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:17.477980 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:17.477983 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:17.477985 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:17.478224 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:17.478820 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:17.478826 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:17.479260 (+ 434us) client_negotiation.cc:769] Sending connection context
0619 14:32:17.479492 (+ 232us) client_negotiation.cc:240] Negotiation successful
0619 14:32:17.479876 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":133,"threads_started":1}
I20260619 14:32:17.482476 5321 heartbeater.cc:347] Connected to a master server at 127.2.2.62:33859
I20260619 14:32:17.482738 5321 heartbeater.cc:464] Registering TS with master...
I20260619 14:32:17.483264 5321 heartbeater.cc:511] Master 127.2.2.62:33859 requested a full tablet report, sending...
I20260619 14:32:17.484958 5130 ts_manager.cc:194] Registered new tserver with Master: 814e118005e8441cacedd828aec64f2d (127.2.2.1:38497)
I20260619 14:32:17.486757 5130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.2.1@KRBTEST.COM'} at 127.2.2.1:35727
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab.
Entry for principal kudu/127.2.2.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab.
I20260619 14:32:17.496731 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.2:0
--local_ip_for_outbound_sockets=127.2.2.2
--webserver_interface=127.2.2.2
--webserver_port=0
--tserver_master_addrs=127.2.2.62:33859
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab
--principal=kudu/127.2.2.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:34685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:32:17.611984 5326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:17.612347 5326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:17.612426 5326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:17.616967 5326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:17.617055 5326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:17.617218 5326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.2
I20260619 14:32:17.623128 5326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:34685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab
--principal=kudu/127.2.2.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.2.2
--webserver_port=0
--tserver_master_addrs=127.2.2.62:33859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.2
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:32:17.624233 5326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:17.625288 5326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:32:17.635098 5332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:17.635133 5326 server_base.cc:1034] running on GCE node
W20260619 14:32:17.635092 5331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:17.635314 5334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:17.635751 5326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:17.636479 5326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:17.637682 5326 hybrid_clock.cc:648] HybridClock initialized: now 1781879537637671 us; error 25 us; skew 500 ppm
Jun 19 14:32:17 dist-test-slave-70rm krb5kdc[5085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879537, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:32:17.640897 5326 init.cc:377] Logged in from keytab as kudu/127.2.2.2@KRBTEST.COM (short username kudu)
I20260619 14:32:17.641795 5326 webserver.cc:460] Webserver started at http://127.2.2.2:40961/ using document root <none> and password file <none>
I20260619 14:32:17.642431 5326 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:17.642522 5326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:17.642771 5326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:32:17.644932 5326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-1/data/instance:
uuid: "5e4f6d74a353470d854d39f113c05053"
format_stamp: "Formatted at 2026-06-19 14:32:17 on dist-test-slave-70rm"
server_key: "e7a02432d072ca82d4fcb8a2ef3d3767"
server_key_iv: "26025456a48b5df2ecebb5ec294bffbf"
server_key_version: "encryptionkey@0"
I20260619 14:32:17.645577 5326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-1/wal/instance:
uuid: "5e4f6d74a353470d854d39f113c05053"
format_stamp: "Formatted at 2026-06-19 14:32:17 on dist-test-slave-70rm"
server_key: "e7a02432d072ca82d4fcb8a2ef3d3767"
server_key_iv: "26025456a48b5df2ecebb5ec294bffbf"
server_key_version: "encryptionkey@0"
I20260619 14:32:17.649367 5326 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 14:32:17.652097 5341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:17.653347 5326 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:32:17.653487 5326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-1/wal
uuid: "5e4f6d74a353470d854d39f113c05053"
format_stamp: "Formatted at 2026-06-19 14:32:17 on dist-test-slave-70rm"
server_key: "e7a02432d072ca82d4fcb8a2ef3d3767"
server_key_iv: "26025456a48b5df2ecebb5ec294bffbf"
server_key_version: "encryptionkey@0"
I20260619 14:32:17.653607 5326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:17.664429 5326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:17.667869 5326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:32:17.668103 5326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:17.668756 5326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:32:17.669663 5326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:32:17.669746 5326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:17.669821 5326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:32:17.669857 5326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:17.679804 5326 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.2:45087
I20260619 14:32:17.679821 5454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.2:45087 every 8 connection(s)
I20260619 14:32:17.680768 5326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
I20260619 14:32:17.683574 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 5326
I20260619 14:32:17.683701 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-1/wal/instance
I20260619 14:32:17.684015 2056 external_mini_cluster.cc:1428] Setting key cd8a0e18fa58e0a8fed69288c5171d4d
Jun 19 14:32:17 dist-test-slave-70rm krb5kdc[5085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879537, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.2@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.2.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.3@KRBTEST.COM" created.
I20260619 14:32:17.695721 5169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:17.682736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:52647 (local address 127.2.2.62:33859)
0619 14:32:17.682881 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:17.682885 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:17.683835 (+ 950us) server_negotiation.cc:407] Connection header received
0619 14:32:17.685469 (+ 1634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:17.685472 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:17.685523 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:17.685621 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:17.686930 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:17.687473 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:17.688660 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:17.688894 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:17.689007 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:17.691912 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:17.691916 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:17.691918 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:17.691944 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:17.693506 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:17.694043 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:17.694047 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:17.694049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:17.694197 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:17.694557 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:17.694562 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:17.694564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:17.694762 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:17.694868 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:17.695421 (+ 553us) server_negotiation.cc:299] Negotiation successful
0619 14:32:17.695553 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 14:32:17.696911 5457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:17.683083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:33859 (local address 127.2.2.2:52647)
0619 14:32:17.683612 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:17.683708 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:17.685213 (+ 1505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:17.685811 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:17.685819 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:17.686268 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:17.686781 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:17.686795 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:17.687608 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:17.687617 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:17.688523 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:17.688532 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:17.688990 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:17.689052 (+ 62us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:17.689232 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:17.689873 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:17.689890 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:17.691720 (+ 1830us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:17.693627 (+ 1907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:17.693642 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:17.693660 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:17.693933 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:17.694294 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:17.694297 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:17.694299 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:17.694459 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:17.694876 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:17.694886 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:17.695179 (+ 293us) client_negotiation.cc:769] Sending connection context
0619 14:32:17.695382 (+ 203us) client_negotiation.cc:240] Negotiation successful
0619 14:32:17.695761 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":157,"threads_started":1}
I20260619 14:32:17.698266 5455 heartbeater.cc:347] Connected to a master server at 127.2.2.62:33859
I20260619 14:32:17.698544 5455 heartbeater.cc:464] Registering TS with master...
I20260619 14:32:17.699040 5455 heartbeater.cc:511] Master 127.2.2.62:33859 requested a full tablet report, sending...
I20260619 14:32:17.700121 5130 ts_manager.cc:194] Registered new tserver with Master: 5e4f6d74a353470d854d39f113c05053 (127.2.2.2:45087)
I20260619 14:32:17.700747 5130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.2.2@KRBTEST.COM'} at 127.2.2.2:52647
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab.
Entry for principal kudu/127.2.2.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab.
I20260619 14:32:17.713660 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.3:0
--local_ip_for_outbound_sockets=127.2.2.3
--webserver_interface=127.2.2.3
--webserver_port=0
--tserver_master_addrs=127.2.2.62:33859
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab
--principal=kudu/127.2.2.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:34685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:32:17.826313 5460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:17.826613 5460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:17.826659 5460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:17.830775 5460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:17.830883 5460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:17.830983 5460 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.3
I20260619 14:32:17.836431 5460 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:34685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab
--principal=kudu/127.2.2.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.2.3
--webserver_port=0
--tserver_master_addrs=127.2.2.62:33859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.3
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:32:17.837600 5460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:17.838709 5460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:32:17.848829 5466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:17.848940 5465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:17.848867 5460 server_base.cc:1034] running on GCE node
W20260619 14:32:17.849138 5468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:17.849560 5460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:17.850260 5460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:17.851457 5460 hybrid_clock.cc:648] HybridClock initialized: now 1781879537851434 us; error 35 us; skew 500 ppm
Jun 19 14:32:17 dist-test-slave-70rm krb5kdc[5085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879537, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:32:17.854501 5460 init.cc:377] Logged in from keytab as kudu/127.2.2.3@KRBTEST.COM (short username kudu)
I20260619 14:32:17.855440 5460 webserver.cc:460] Webserver started at http://127.2.2.3:40799/ using document root <none> and password file <none>
I20260619 14:32:17.856031 5460 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:17.856129 5460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:17.856377 5460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:32:17.858752 5460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-2/data/instance:
uuid: "d77dee2322274c298ccc2fc736c4d58f"
format_stamp: "Formatted at 2026-06-19 14:32:17 on dist-test-slave-70rm"
server_key: "089e540c5d329658479a4d80223b3ebe"
server_key_iv: "44c0235c350f09350c7e5df620338c51"
server_key_version: "encryptionkey@0"
I20260619 14:32:17.859339 5460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-2/wal/instance:
uuid: "d77dee2322274c298ccc2fc736c4d58f"
format_stamp: "Formatted at 2026-06-19 14:32:17 on dist-test-slave-70rm"
server_key: "089e540c5d329658479a4d80223b3ebe"
server_key_iv: "44c0235c350f09350c7e5df620338c51"
server_key_version: "encryptionkey@0"
I20260619 14:32:17.863219 5460 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:32:17.865803 5475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:17.867221 5460 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:32:17.867334 5460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-2/wal
uuid: "d77dee2322274c298ccc2fc736c4d58f"
format_stamp: "Formatted at 2026-06-19 14:32:17 on dist-test-slave-70rm"
server_key: "089e540c5d329658479a4d80223b3ebe"
server_key_iv: "44c0235c350f09350c7e5df620338c51"
server_key_version: "encryptionkey@0"
I20260619 14:32:17.867427 5460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:17.888181 5460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:17.891849 5460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:32:17.892022 5460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:17.892587 5460 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:32:17.893402 5460 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:32:17.893450 5460 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:17.893498 5460 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:32:17.893517 5460 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:17.902283 5460 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.3:38943
I20260619 14:32:17.902305 5588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.3:38943 every 8 connection(s)
I20260619 14:32:17.903368 5460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
I20260619 14:32:17.910470 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 5460
I20260619 14:32:17.910604 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879514993456-2056-0/minicluster-data/ts-2/wal/instance
I20260619 14:32:17.910902 2056 external_mini_cluster.cc:1428] Setting key 22b47e267718bc726db067aa08111494
Jun 19 14:32:17 dist-test-slave-70rm krb5kdc[5085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879537, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.3@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:32:17.917377 5169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:17.905288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.3:35847 (local address 127.2.2.62:33859)
0619 14:32:17.905442 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:17.905447 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:17.906463 (+ 1016us) server_negotiation.cc:407] Connection header received
0619 14:32:17.907770 (+ 1307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:17.907772 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:17.907818 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:17.907893 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:17.909107 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:17.909634 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:17.911027 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:17.911318 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:17.911441 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:17.914317 (+ 2876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:17.914320 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:17.914322 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:17.914344 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:17.915591 (+ 1247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:17.916074 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:17.916076 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:17.916077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:17.916163 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:17.916445 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:17.916448 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:17.916449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:17.916606 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:17.916674 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:17.917167 (+ 493us) server_negotiation.cc:299] Negotiation successful
0619 14:32:17.917262 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 14:32:17.918246 5591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:17.905735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:33859 (local address 127.2.2.3:35847)
0619 14:32:17.906304 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:17.906350 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:17.907574 (+ 1224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:17.908092 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:17.908099 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:17.908453 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:17.908940 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:17.908953 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:17.909742 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:17.909745 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:17.910875 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:17.910885 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:17.911446 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:17.911496 (+ 50us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:17.911698 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:17.912430 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:17.912451 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:17.914093 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:17.915691 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:17.915696 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:17.915710 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:17.915962 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:17.916236 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:17.916238 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:17.916240 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:17.916365 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:17.916686 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:17.916691 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:17.916922 (+ 231us) client_negotiation.cc:769] Sending connection context
0619 14:32:17.917098 (+ 176us) client_negotiation.cc:240] Negotiation successful
0619 14:32:17.917349 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":111,"threads_started":1}
I20260619 14:32:17.919413 5589 heartbeater.cc:347] Connected to a master server at 127.2.2.62:33859
I20260619 14:32:17.919682 5589 heartbeater.cc:464] Registering TS with master...
I20260619 14:32:17.920159 5589 heartbeater.cc:511] Master 127.2.2.62:33859 requested a full tablet report, sending...
I20260619 14:32:17.921195 5130 ts_manager.cc:194] Registered new tserver with Master: d77dee2322274c298ccc2fc736c4d58f (127.2.2.3:38943)
I20260619 14:32:17.921838 5130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.2.3@KRBTEST.COM'} at 127.2.2.3:35847
I20260619 14:32:17.924646 2056 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 14:32:17 dist-test-slave-70rm krb5kdc[5085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879537, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 19 14:32:17 dist-test-slave-70rm krb5kdc[5085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879537, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:32:17.950784 5169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:17.941903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42248 (local address 127.2.2.62:33859)
0619 14:32:17.942075 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:17.942080 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:17.942306 (+ 226us) server_negotiation.cc:407] Connection header received
0619 14:32:17.942486 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:17.942489 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:17.942531 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:17.942623 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:17.943336 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:17.943865 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:17.944796 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:17.945008 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:17.945121 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:17.947706 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:17.947715 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:17.947717 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:17.947743 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:17.948989 (+ 1246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:17.949448 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:17.949452 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:17.949454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:17.949536 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:17.949827 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:17.949831 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:17.949833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:17.950064 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:17.950227 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:17.950476 (+ 249us) server_negotiation.cc:299] Negotiation successful
0619 14:32:17.950596 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260619 14:32:17.951224 5130 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:42248
I20260619 14:32:17.952214 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 5192
I20260619 14:32:17.957581 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 5326
I20260619 14:32:17.962810 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 5460
I20260619 14:32:17.967749 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 5098
2026-06-19T14:32:17Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2149 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:32:18 dist-test-slave-70rm krb5kdc[5601](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:32:18 dist-test-slave-70rm krb5kdc[5601](info): set up 2 sockets
Jun 19 14:32:18 dist-test-slave-70rm krb5kdc[5601](info): commencing operation
krb5kdc: starting...
W20260619 14:32:19.002296 2056 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Jun 19 14:32:19 dist-test-slave-70rm krb5kdc[5601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:32:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:32:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.2.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab.
Entry for principal kudu/127.2.2.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab.
I20260619 14:32:19.125146 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:41241
--webserver_interface=127.2.2.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:35957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.2.62:41241
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:32:19.235628 5614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:19.235921 5614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:19.235963 5614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:19.240118 5614 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:32:19.240183 5614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:19.240203 5614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:19.240221 5614 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:32:19.240236 5614 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:32:19.245785 5614 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:35957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.2.62:41241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:41241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.2.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:32:19.246961 5614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:19.248064 5614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:32:19.256814 5622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:19.256816 5620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:19.256853 5619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:19.257081 5614 server_base.cc:1034] running on GCE node
I20260619 14:32:19.257758 5614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:19.259037 5614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:19.260226 5614 hybrid_clock.cc:648] HybridClock initialized: now 1781879539260212 us; error 30 us; skew 500 ppm
Jun 19 14:32:19 dist-test-slave-70rm krb5kdc[5601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879539, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:32:19.263120 5614 init.cc:377] Logged in from keytab as kudu/127.2.2.62@KRBTEST.COM (short username kudu)
I20260619 14:32:19.264070 5614 webserver.cc:460] Webserver started at http://127.2.2.62:44589/ using document root <none> and password file <none>
I20260619 14:32:19.264647 5614 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:19.264712 5614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:19.264953 5614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:32:19.267117 5614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/master-0/data/instance:
uuid: "7b09bf0a13224bddbbccb6c3a4ab4053"
format_stamp: "Formatted at 2026-06-19 14:32:19 on dist-test-slave-70rm"
server_key: "bbcc9bfbdb7f1d922376156c144179ea"
server_key_iv: "975498debe079c516bd1c138446edfd7"
server_key_version: "encryptionkey@0"
I20260619 14:32:19.267697 5614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/master-0/wal/instance:
uuid: "7b09bf0a13224bddbbccb6c3a4ab4053"
format_stamp: "Formatted at 2026-06-19 14:32:19 on dist-test-slave-70rm"
server_key: "bbcc9bfbdb7f1d922376156c144179ea"
server_key_iv: "975498debe079c516bd1c138446edfd7"
server_key_version: "encryptionkey@0"
I20260619 14:32:19.271510 5614 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:32:19.273967 5629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:19.275261 5614 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:32:19.275408 5614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/master-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/master-0/wal
uuid: "7b09bf0a13224bddbbccb6c3a4ab4053"
format_stamp: "Formatted at 2026-06-19 14:32:19 on dist-test-slave-70rm"
server_key: "bbcc9bfbdb7f1d922376156c144179ea"
server_key_iv: "975498debe079c516bd1c138446edfd7"
server_key_version: "encryptionkey@0"
I20260619 14:32:19.275519 5614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:19.295020 5614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:19.298077 5614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:32:19.298354 5614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:19.305554 5614 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.62:41241
I20260619 14:32:19.305567 5681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.62:41241 every 8 connection(s)
I20260619 14:32:19.306581 5614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
I20260619 14:32:19.309329 5682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:19.311321 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 5614
I20260619 14:32:19.311494 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/master-0/wal/instance
I20260619 14:32:19.311792 2056 external_mini_cluster.cc:1428] Setting key 91e6b1d1f15537b8095c3f463e6b53c0
I20260619 14:32:19.316094 5682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b09bf0a13224bddbbccb6c3a4ab4053: Bootstrap starting.
I20260619 14:32:19.318691 5682 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b09bf0a13224bddbbccb6c3a4ab4053: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:19.319602 5682 log.cc:824] T 00000000000000000000000000000000 P 7b09bf0a13224bddbbccb6c3a4ab4053: Log is configured to *not* fsync() on all Append() calls
Jun 19 14:32:19 dist-test-slave-70rm krb5kdc[5601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:32:19.321800 5682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b09bf0a13224bddbbccb6c3a4ab4053: No bootstrap required, opened a new log
I20260619 14:32:19.326570 5685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:19.313126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47046 (local address 127.2.2.62:41241)
0619 14:32:19.313535 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:19.313550 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:19.313585 (+ 35us) server_negotiation.cc:407] Connection header received
0619 14:32:19.314287 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:19.314305 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:19.314645 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:19.314926 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:19.315810 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:19.316849 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:19.318023 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:19.318563 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:19.318684 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:19.321319 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:19.321325 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:19.321341 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:19.321371 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:19.323401 (+ 2030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:19.323824 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:19.323830 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:19.323836 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:19.323976 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:19.324237 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:19.324241 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:19.324243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:19.324821 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:19.324991 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:19.325225 (+ 234us) server_negotiation.cc:299] Negotiation successful
0619 14:32:19.325544 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":135,"threads_started":1}
I20260619 14:32:19.326660 5682 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b09bf0a13224bddbbccb6c3a4ab4053 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b09bf0a13224bddbbccb6c3a4ab4053" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41241 } }
I20260619 14:32:19.326920 5682 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b09bf0a13224bddbbccb6c3a4ab4053 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:19.326973 5682 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b09bf0a13224bddbbccb6c3a4ab4053 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b09bf0a13224bddbbccb6c3a4ab4053, State: Initialized, Role: FOLLOWER
I20260619 14:32:19.327443 5682 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b09bf0a13224bddbbccb6c3a4ab4053 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b09bf0a13224bddbbccb6c3a4ab4053" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41241 } }
I20260619 14:32:19.327602 5682 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b09bf0a13224bddbbccb6c3a4ab4053 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:32:19.327684 5682 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b09bf0a13224bddbbccb6c3a4ab4053 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:32:19.327785 5682 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b09bf0a13224bddbbccb6c3a4ab4053 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:19.328791 5682 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b09bf0a13224bddbbccb6c3a4ab4053 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b09bf0a13224bddbbccb6c3a4ab4053" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41241 } }
I20260619 14:32:19.329118 5682 leader_election.cc:302] T 00000000000000000000000000000000 P 7b09bf0a13224bddbbccb6c3a4ab4053 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b09bf0a13224bddbbccb6c3a4ab4053; no voters:
I20260619 14:32:19.329380 5682 leader_election.cc:288] T 00000000000000000000000000000000 P 7b09bf0a13224bddbbccb6c3a4ab4053 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:32:19.329505 5687 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b09bf0a13224bddbbccb6c3a4ab4053 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:32:19.329731 5687 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b09bf0a13224bddbbccb6c3a4ab4053 [term 1 LEADER]: Becoming Leader. State: Replica: 7b09bf0a13224bddbbccb6c3a4ab4053, State: Running, Role: LEADER
I20260619 14:32:19.330029 5687 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b09bf0a13224bddbbccb6c3a4ab4053 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b09bf0a13224bddbbccb6c3a4ab4053" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41241 } }
I20260619 14:32:19.330698 5682 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b09bf0a13224bddbbccb6c3a4ab4053 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:32:19.331696 5688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b09bf0a13224bddbbccb6c3a4ab4053 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b09bf0a13224bddbbccb6c3a4ab4053" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b09bf0a13224bddbbccb6c3a4ab4053" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41241 } } }
I20260619 14:32:19.331840 5688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b09bf0a13224bddbbccb6c3a4ab4053 [sys.catalog]: This master's current role is: LEADER
I20260619 14:32:19.332240 5696 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:32:19.332192 5689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b09bf0a13224bddbbccb6c3a4ab4053 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b09bf0a13224bddbbccb6c3a4ab4053. Latest consensus state: current_term: 1 leader_uuid: "7b09bf0a13224bddbbccb6c3a4ab4053" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b09bf0a13224bddbbccb6c3a4ab4053" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41241 } } }
I20260619 14:32:19.332410 5689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b09bf0a13224bddbbccb6c3a4ab4053 [sys.catalog]: This master's current role is: LEADER
I20260619 14:32:19.335475 5696 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:32:19.341190 5696 catalog_manager.cc:1347] Generated new cluster ID: c53ab62061f84cad9bc88db9c176000f
I20260619 14:32:19.341285 5696 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:32:19.355530 5696 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:32:19.356854 5696 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:32:19.363711 5696 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b09bf0a13224bddbbccb6c3a4ab4053: Generated new TSK 0
I20260619 14:32:19.369124 5696 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.2.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab.
Entry for principal kudu/127.2.2.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab.
I20260619 14:32:19.397709 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.1:0
--local_ip_for_outbound_sockets=127.2.2.1
--webserver_interface=127.2.2.1
--webserver_port=0
--tserver_master_addrs=127.2.2.62:41241
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab
--principal=kudu/127.2.2.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:35957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:32:19.511745 5708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:19.512042 5708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:19.512118 5708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:19.516153 5708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:19.516217 5708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:19.516317 5708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.1
I20260619 14:32:19.521611 5708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:35957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab
--principal=kudu/127.2.2.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.2.1
--webserver_port=0
--tserver_master_addrs=127.2.2.62:41241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.1
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:32:19.522750 5708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:19.523833 5708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:32:19.533980 5713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:19.533977 5714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:19.533968 5716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:19.534466 5708 server_base.cc:1034] running on GCE node
I20260619 14:32:19.534880 5708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:19.535620 5708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:19.536830 5708 hybrid_clock.cc:648] HybridClock initialized: now 1781879539536807 us; error 40 us; skew 500 ppm
Jun 19 14:32:19 dist-test-slave-70rm krb5kdc[5601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879539, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:32:19.539847 5708 init.cc:377] Logged in from keytab as kudu/127.2.2.1@KRBTEST.COM (short username kudu)
I20260619 14:32:19.540807 5708 webserver.cc:460] Webserver started at http://127.2.2.1:34199/ using document root <none> and password file <none>
I20260619 14:32:19.541391 5708 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:19.541482 5708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:19.541728 5708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:32:19.543954 5708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/ts-0/data/instance:
uuid: "c940b9610f0f4319b3ea415fd2f373eb"
format_stamp: "Formatted at 2026-06-19 14:32:19 on dist-test-slave-70rm"
server_key: "43744cf5bda6d77d847fdab6a0fdc562"
server_key_iv: "9b1640fea07b94339d33d71a3f456001"
server_key_version: "encryptionkey@0"
I20260619 14:32:19.544574 5708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/ts-0/wal/instance:
uuid: "c940b9610f0f4319b3ea415fd2f373eb"
format_stamp: "Formatted at 2026-06-19 14:32:19 on dist-test-slave-70rm"
server_key: "43744cf5bda6d77d847fdab6a0fdc562"
server_key_iv: "9b1640fea07b94339d33d71a3f456001"
server_key_version: "encryptionkey@0"
I20260619 14:32:19.548759 5708 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:32:19.551523 5723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:19.552891 5708 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 14:32:19.553053 5708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/ts-0/wal
uuid: "c940b9610f0f4319b3ea415fd2f373eb"
format_stamp: "Formatted at 2026-06-19 14:32:19 on dist-test-slave-70rm"
server_key: "43744cf5bda6d77d847fdab6a0fdc562"
server_key_iv: "9b1640fea07b94339d33d71a3f456001"
server_key_version: "encryptionkey@0"
I20260619 14:32:19.553332 5708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:19.564209 5708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:19.567412 5708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:32:19.567637 5708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:19.568243 5708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:32:19.569098 5708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:32:19.569180 5708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:19.569271 5708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:32:19.569322 5708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:19.578461 5708 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.1:36491
I20260619 14:32:19.578552 5836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.1:36491 every 8 connection(s)
I20260619 14:32:19.579478 5708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
I20260619 14:32:19.584412 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 5708
I20260619 14:32:19.584542 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/ts-0/wal/instance
I20260619 14:32:19.584877 2056 external_mini_cluster.cc:1428] Setting key 695e66df978cfd57ae55f09c8ad7ef48
Jun 19 14:32:19 dist-test-slave-70rm krb5kdc[5601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879539, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.1@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.2.2@KRBTEST.COM; defaulting to no policy
I20260619 14:32:19.597847 5685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:19.581766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:51725 (local address 127.2.2.62:41241)
0619 14:32:19.581960 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:19.581967 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:19.584273 (+ 2306us) server_negotiation.cc:407] Connection header received
0619 14:32:19.586228 (+ 1955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:19.586232 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:19.586289 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:19.586346 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:19.588344 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:19.589119 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:19.590417 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:19.590865 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:19.590999 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:19.593970 (+ 2971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:19.593973 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:19.593975 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:19.594002 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:19.595572 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:19.596229 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:19.596232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:19.596234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:19.596316 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:19.596666 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:19.596669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:19.596670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:19.596852 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:19.596933 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:19.597605 (+ 672us) server_negotiation.cc:299] Negotiation successful
0619 14:32:19.597722 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.2@KRBTEST.COM" created.
I20260619 14:32:19.599017 5839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:19.582418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41241 (local address 127.2.2.1:51725)
0619 14:32:19.584090 (+ 1672us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:19.584155 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:19.585904 (+ 1749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:19.586836 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:19.586848 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:19.587394 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:19.588150 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:19.588174 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:19.589256 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:19.589259 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:19.590292 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:19.590302 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:19.591001 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:19.591053 (+ 52us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:19.591222 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:19.591959 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:19.591980 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:19.593749 (+ 1769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:19.595708 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:19.595715 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:19.595733 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:19.596120 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:19.596424 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:19.596427 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:19.596428 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:19.596580 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:19.596973 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:19.596979 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:19.597370 (+ 391us) client_negotiation.cc:769] Sending connection context
0619 14:32:19.597548 (+ 178us) client_negotiation.cc:240] Negotiation successful
0619 14:32:19.597836 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1424,"thread_start_us":181,"threads_started":1}
I20260619 14:32:19.600330 5837 heartbeater.cc:347] Connected to a master server at 127.2.2.62:41241
I20260619 14:32:19.600600 5837 heartbeater.cc:464] Registering TS with master...
I20260619 14:32:19.601122 5837 heartbeater.cc:511] Master 127.2.2.62:41241 requested a full tablet report, sending...
I20260619 14:32:19.602790 5646 ts_manager.cc:194] Registered new tserver with Master: c940b9610f0f4319b3ea415fd2f373eb (127.2.2.1:36491)
I20260619 14:32:19.604477 5646 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.2.1@KRBTEST.COM'} at 127.2.2.1:51725
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab.
Entry for principal kudu/127.2.2.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab.
I20260619 14:32:19.617303 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.2:0
--local_ip_for_outbound_sockets=127.2.2.2
--webserver_interface=127.2.2.2
--webserver_port=0
--tserver_master_addrs=127.2.2.62:41241
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab
--principal=kudu/127.2.2.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:35957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:32:19.730399 5842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:19.730790 5842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:19.730881 5842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:19.735318 5842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:19.735407 5842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:19.735502 5842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.2
I20260619 14:32:19.740954 5842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:35957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab
--principal=kudu/127.2.2.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.2.2
--webserver_port=0
--tserver_master_addrs=127.2.2.62:41241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.2
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:32:19.742107 5842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:19.743242 5842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:32:19.753191 5847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:19.753209 5848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:19.753185 5850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:19.753247 5842 server_base.cc:1034] running on GCE node
I20260619 14:32:19.753960 5842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:19.754695 5842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:19.755888 5842 hybrid_clock.cc:648] HybridClock initialized: now 1781879539755874 us; error 32 us; skew 500 ppm
Jun 19 14:32:19 dist-test-slave-70rm krb5kdc[5601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879539, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:32:19.758841 5842 init.cc:377] Logged in from keytab as kudu/127.2.2.2@KRBTEST.COM (short username kudu)
I20260619 14:32:19.759807 5842 webserver.cc:460] Webserver started at http://127.2.2.2:41925/ using document root <none> and password file <none>
I20260619 14:32:19.760383 5842 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:19.760471 5842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:19.760713 5842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:32:19.762862 5842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/ts-1/data/instance:
uuid: "ef7f6029607740da9dfa9e76a905e637"
format_stamp: "Formatted at 2026-06-19 14:32:19 on dist-test-slave-70rm"
server_key: "101a24a42cbcca67bbcff3144b26ab00"
server_key_iv: "c1e1b465a1404fd00c8dbe7311673aea"
server_key_version: "encryptionkey@0"
I20260619 14:32:19.763438 5842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/ts-1/wal/instance:
uuid: "ef7f6029607740da9dfa9e76a905e637"
format_stamp: "Formatted at 2026-06-19 14:32:19 on dist-test-slave-70rm"
server_key: "101a24a42cbcca67bbcff3144b26ab00"
server_key_iv: "c1e1b465a1404fd00c8dbe7311673aea"
server_key_version: "encryptionkey@0"
I20260619 14:32:19.767181 5842 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 14:32:19.769721 5857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:19.771095 5842 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 14:32:19.771231 5842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/ts-1/wal
uuid: "ef7f6029607740da9dfa9e76a905e637"
format_stamp: "Formatted at 2026-06-19 14:32:19 on dist-test-slave-70rm"
server_key: "101a24a42cbcca67bbcff3144b26ab00"
server_key_iv: "c1e1b465a1404fd00c8dbe7311673aea"
server_key_version: "encryptionkey@0"
I20260619 14:32:19.771345 5842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:19.783082 5842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:19.785938 5842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:32:19.786190 5842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:19.786778 5842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:32:19.787582 5842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:32:19.787652 5842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:19.787721 5842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:32:19.787771 5842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:19.797807 5842 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.2:40549
I20260619 14:32:19.797823 5970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.2:40549 every 8 connection(s)
I20260619 14:32:19.798839 5842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
I20260619 14:32:19.803789 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 5842
I20260619 14:32:19.803885 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/ts-1/wal/instance
I20260619 14:32:19.804175 2056 external_mini_cluster.cc:1428] Setting key 3a300e8e0696e04d91e5d93e610c812a
Jun 19 14:32:19 dist-test-slave-70rm krb5kdc[5601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879539, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.2@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.2.3@KRBTEST.COM; defaulting to no policy
I20260619 14:32:19.815248 5685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:19.800890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:33957 (local address 127.2.2.62:41241)
0619 14:32:19.801053 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:19.801057 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:19.801906 (+ 849us) server_negotiation.cc:407] Connection header received
0619 14:32:19.803414 (+ 1508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:19.803418 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:19.803473 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:19.803554 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:19.804932 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:19.805671 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:19.807560 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:19.807950 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:19.808068 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:19.811099 (+ 3031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:19.811105 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:19.811111 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:19.811149 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:19.812911 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:19.813498 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:19.813504 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:19.813505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:19.813602 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:19.813976 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:19.813980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:19.813984 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:19.814282 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:19.814397 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:19.814961 (+ 564us) server_negotiation.cc:299] Negotiation successful
0619 14:32:19.815063 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 14:32:19.816341 5973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:19.801313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41241 (local address 127.2.2.2:33957)
0619 14:32:19.801752 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:19.801796 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:19.803191 (+ 1395us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:19.803728 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:19.803735 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:19.804186 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:19.804758 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:19.804776 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:19.805811 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:19.805815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:19.807408 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:19.807423 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:19.808104 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:19.808152 (+ 48us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:19.808299 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:19.808953 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:19.808970 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:19.810864 (+ 1894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:19.813056 (+ 2192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:19.813061 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:19.813076 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:19.813354 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:19.813708 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:19.813711 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:19.813712 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:19.813868 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:19.814425 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:19.814430 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:19.814715 (+ 285us) client_negotiation.cc:769] Sending connection context
0619 14:32:19.814902 (+ 187us) client_negotiation.cc:240] Negotiation successful
0619 14:32:19.815307 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":115,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.3@KRBTEST.COM" created.
I20260619 14:32:19.817713 5971 heartbeater.cc:347] Connected to a master server at 127.2.2.62:41241
I20260619 14:32:19.817981 5971 heartbeater.cc:464] Registering TS with master...
I20260619 14:32:19.818534 5971 heartbeater.cc:511] Master 127.2.2.62:41241 requested a full tablet report, sending...
I20260619 14:32:19.819621 5646 ts_manager.cc:194] Registered new tserver with Master: ef7f6029607740da9dfa9e76a905e637 (127.2.2.2:40549)
I20260619 14:32:19.820497 5646 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.2.2@KRBTEST.COM'} at 127.2.2.2:33957
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab.
Entry for principal kudu/127.2.2.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab.
I20260619 14:32:19.832873 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.3:0
--local_ip_for_outbound_sockets=127.2.2.3
--webserver_interface=127.2.2.3
--webserver_port=0
--tserver_master_addrs=127.2.2.62:41241
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab
--principal=kudu/127.2.2.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:35957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:32:19.944233 5976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:19.944514 5976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:19.944557 5976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:19.948781 5976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:19.948848 5976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:19.948956 5976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.3
I20260619 14:32:19.954595 5976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:35957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab
--principal=kudu/127.2.2.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.2.3
--webserver_port=0
--tserver_master_addrs=127.2.2.62:41241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.3
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:32:19.955628 5976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:19.956669 5976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:32:19.966512 5981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:19.966605 5976 server_base.cc:1034] running on GCE node
W20260619 14:32:19.966555 5982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:19.966517 5984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:19.967234 5976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:19.967902 5976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:19.969070 5976 hybrid_clock.cc:648] HybridClock initialized: now 1781879539969045 us; error 32 us; skew 500 ppm
Jun 19 14:32:19 dist-test-slave-70rm krb5kdc[5601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879539, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:32:19.971885 5976 init.cc:377] Logged in from keytab as kudu/127.2.2.3@KRBTEST.COM (short username kudu)
I20260619 14:32:19.972759 5976 webserver.cc:460] Webserver started at http://127.2.2.3:45303/ using document root <none> and password file <none>
I20260619 14:32:19.973274 5976 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:19.973333 5976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:19.973496 5976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:32:19.975766 5976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/ts-2/data/instance:
uuid: "177797fdf8a54b758ae99065dfab6128"
format_stamp: "Formatted at 2026-06-19 14:32:19 on dist-test-slave-70rm"
server_key: "91c885239a9c5df974ca33049b83aaab"
server_key_iv: "292c31950fc69fbc12b4ffb6318f6042"
server_key_version: "encryptionkey@0"
I20260619 14:32:19.976289 5976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/ts-2/wal/instance:
uuid: "177797fdf8a54b758ae99065dfab6128"
format_stamp: "Formatted at 2026-06-19 14:32:19 on dist-test-slave-70rm"
server_key: "91c885239a9c5df974ca33049b83aaab"
server_key_iv: "292c31950fc69fbc12b4ffb6318f6042"
server_key_version: "encryptionkey@0"
I20260619 14:32:19.980309 5976 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:32:19.982817 5991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:19.984174 5976 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:32:19.984301 5976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/ts-2/wal
uuid: "177797fdf8a54b758ae99065dfab6128"
format_stamp: "Formatted at 2026-06-19 14:32:19 on dist-test-slave-70rm"
server_key: "91c885239a9c5df974ca33049b83aaab"
server_key_iv: "292c31950fc69fbc12b4ffb6318f6042"
server_key_version: "encryptionkey@0"
I20260619 14:32:19.984419 5976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:19.997047 5976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:20.000293 5976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:32:20.000501 5976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:20.001106 5976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:32:20.001950 5976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:32:20.001997 5976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:20.002064 5976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:32:20.002097 5976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:20.011170 5976 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.3:41867
I20260619 14:32:20.011196 6104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.3:41867 every 8 connection(s)
I20260619 14:32:20.012105 5976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
I20260619 14:32:20.019145 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 5976
I20260619 14:32:20.019268 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879514993456-2056-0/minicluster-data/ts-2/wal/instance
I20260619 14:32:20.019541 2056 external_mini_cluster.cc:1428] Setting key bbe2af09b0b677d35ee0192eb1a98081
Jun 19 14:32:20 dist-test-slave-70rm krb5kdc[5601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879539, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.3@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:32:20.025969 5685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:20.013961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.3:49581 (local address 127.2.2.62:41241)
0619 14:32:20.014149 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:20.014155 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:20.014981 (+ 826us) server_negotiation.cc:407] Connection header received
0619 14:32:20.016324 (+ 1343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:20.016327 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:20.016374 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:20.016448 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:20.017614 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:20.018179 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:20.019415 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:20.019711 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:20.019825 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:20.022701 (+ 2876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:20.022704 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:20.022706 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:20.022731 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:20.023989 (+ 1258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:20.024487 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:20.024490 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:20.024492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:20.024573 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:20.024903 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:20.024906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:20.024907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:20.025117 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:20.025197 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:20.025721 (+ 524us) server_negotiation.cc:299] Negotiation successful
0619 14:32:20.025837 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 14:32:20.026813 6107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:20.014384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41241 (local address 127.2.2.3:49581)
0619 14:32:20.014832 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:20.014875 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:20.016132 (+ 1257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:20.016616 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:20.016623 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:20.016975 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:20.017491 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:20.017505 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:20.018308 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:20.018311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:20.019207 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:20.019242 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:20.019810 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:20.019866 (+ 56us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:20.020120 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:20.020869 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:20.020891 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:20.022549 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:20.024088 (+ 1539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:20.024092 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:20.024104 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:20.024378 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:20.024661 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:20.024663 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:20.024664 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:20.024811 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:20.025220 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:20.025225 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:20.025468 (+ 243us) client_negotiation.cc:769] Sending connection context
0619 14:32:20.025629 (+ 161us) client_negotiation.cc:240] Negotiation successful
0619 14:32:20.025898 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":120,"threads_started":1}
I20260619 14:32:20.027964 6105 heartbeater.cc:347] Connected to a master server at 127.2.2.62:41241
I20260619 14:32:20.028249 6105 heartbeater.cc:464] Registering TS with master...
I20260619 14:32:20.028750 6105 heartbeater.cc:511] Master 127.2.2.62:41241 requested a full tablet report, sending...
I20260619 14:32:20.029799 5646 ts_manager.cc:194] Registered new tserver with Master: 177797fdf8a54b758ae99065dfab6128 (127.2.2.3:41867)
I20260619 14:32:20.030483 5646 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.2.3@KRBTEST.COM'} at 127.2.2.3:49581
I20260619 14:32:20.033435 2056 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 14:32:20 dist-test-slave-70rm krb5kdc[5601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879540, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 14:32:20 dist-test-slave-70rm krb5kdc[5601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879540, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.2.1@KRBTEST.COM
I20260619 14:32:20.059361 6112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:20.049268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59882 (local address 127.2.2.1:36491)
0619 14:32:20.049612 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:20.049627 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:20.049661 (+ 34us) server_negotiation.cc:407] Connection header received
0619 14:32:20.049710 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:20.049738 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:20.049971 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:20.050128 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:20.050911 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:20.051737 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:20.052649 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:20.053031 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:20.053151 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:20.055683 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:20.055695 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:20.055709 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:20.055734 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:20.057488 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:20.057942 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:20.057947 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:20.057948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:20.058038 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:20.058343 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:20.058346 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:20.058349 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:20.058596 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:20.058850 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:20.059010 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 14:32:20.059172 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":102,"threads_started":1}
Jun 19 14:32:20 dist-test-slave-70rm krb5kdc[5601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879540, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:32:20.070048 5685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:20.062079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47060 (local address 127.2.2.62:41241)
0619 14:32:20.062237 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:20.062240 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:20.062393 (+ 153us) server_negotiation.cc:407] Connection header received
0619 14:32:20.062478 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:20.062480 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:20.062522 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:20.062590 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:20.063213 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:20.063719 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:20.064538 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:20.064738 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:20.064811 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:20.067256 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:20.067259 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:20.067262 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:20.067286 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:20.068698 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:20.069074 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:20.069076 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:20.069078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:20.069147 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:20.069361 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:20.069363 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:20.069364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:20.069568 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:20.069666 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:20.069841 (+ 175us) server_negotiation.cc:299] Negotiation successful
0619 14:32:20.069928 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260619 14:32:20.070468 5646 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47060
I20260619 14:32:20.071066 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 5708
I20260619 14:32:20.076681 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 5842
I20260619 14:32:20.081914 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 5976
I20260619 14:32:20.087214 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 5614
2026-06-19T14:32:20Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2119 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:32:20 dist-test-slave-70rm krb5kdc[6117](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:32:20 dist-test-slave-70rm krb5kdc[6117](info): set up 2 sockets
Jun 19 14:32:20 dist-test-slave-70rm krb5kdc[6117](info): commencing operation
krb5kdc: starting...
W20260619 14:32:21.134649 2056 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Jun 19 14:32:21 dist-test-slave-70rm krb5kdc[6117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:32:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:32:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.2.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab.
Entry for principal kudu/127.2.2.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab.
I20260619 14:32:21.257316 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:42419
--webserver_interface=127.2.2.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:44757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.2.62:42419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:32:21.367461 6130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:21.367802 6130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:21.367880 6130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:21.372094 6130 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:32:21.372187 6130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:21.372225 6130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:21.372284 6130 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:32:21.372325 6130 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:32:21.377653 6130 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:44757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.2.62:42419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:42419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.2.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:32:21.378886 6130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:21.379981 6130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:32:21.388652 6138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:21.388675 6136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:21.388675 6135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:21.388842 6130 server_base.cc:1034] running on GCE node
I20260619 14:32:21.389652 6130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:21.390933 6130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:21.392097 6130 hybrid_clock.cc:648] HybridClock initialized: now 1781879541392065 us; error 52 us; skew 500 ppm
Jun 19 14:32:21 dist-test-slave-70rm krb5kdc[6117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879541, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:32:21.395084 6130 init.cc:377] Logged in from keytab as kudu/127.2.2.62@KRBTEST.COM (short username kudu)
I20260619 14:32:21.395843 6130 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:21.395941 6130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:21.396206 6130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:32:21.398398 6130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/master-0/data/instance:
uuid: "13d21fe2120149a39823873153a21d36"
format_stamp: "Formatted at 2026-06-19 14:32:21 on dist-test-slave-70rm"
server_key: "e848ab70077b0f63828bd2dec89c21a1"
server_key_iv: "41399c735ac335d20c42124e322897e3"
server_key_version: "encryptionkey@0"
I20260619 14:32:21.398983 6130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/master-0/wal/instance:
uuid: "13d21fe2120149a39823873153a21d36"
format_stamp: "Formatted at 2026-06-19 14:32:21 on dist-test-slave-70rm"
server_key: "e848ab70077b0f63828bd2dec89c21a1"
server_key_iv: "41399c735ac335d20c42124e322897e3"
server_key_version: "encryptionkey@0"
I20260619 14:32:21.402864 6130 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 14:32:21.405392 6144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:21.406757 6130 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 14:32:21.406901 6130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/master-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/master-0/wal
uuid: "13d21fe2120149a39823873153a21d36"
format_stamp: "Formatted at 2026-06-19 14:32:21 on dist-test-slave-70rm"
server_key: "e848ab70077b0f63828bd2dec89c21a1"
server_key_iv: "41399c735ac335d20c42124e322897e3"
server_key_version: "encryptionkey@0"
I20260619 14:32:21.407015 6130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:21.438202 6130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:21.441776 6130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:32:21.442003 6130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:21.449260 6130 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.62:42419
I20260619 14:32:21.449543 6196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.62:42419 every 8 connection(s)
I20260619 14:32:21.450014 6130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
I20260619 14:32:21.452904 6197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:21.453912 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 6130
I20260619 14:32:21.454036 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/master-0/wal/instance
I20260619 14:32:21.454355 2056 external_mini_cluster.cc:1428] Setting key c262815a2d512549a8a1f8f4e2b60b8b
I20260619 14:32:21.459318 6197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13d21fe2120149a39823873153a21d36: Bootstrap starting.
I20260619 14:32:21.461522 6197 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13d21fe2120149a39823873153a21d36: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:21.462303 6197 log.cc:824] T 00000000000000000000000000000000 P 13d21fe2120149a39823873153a21d36: Log is configured to *not* fsync() on all Append() calls
Jun 19 14:32:21 dist-test-slave-70rm krb5kdc[6117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:32:21.464417 6197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13d21fe2120149a39823873153a21d36: No bootstrap required, opened a new log
I20260619 14:32:21.468492 6200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:21.455848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55608 (local address 127.2.2.62:42419)
0619 14:32:21.456286 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:21.456303 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:21.456338 (+ 35us) server_negotiation.cc:407] Connection header received
0619 14:32:21.457000 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:21.457017 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:21.457357 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:21.457723 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:21.458525 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:21.459545 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:21.460416 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:21.460901 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:21.461071 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:21.463769 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:21.463774 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:21.463788 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:21.463813 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:21.465599 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:21.466080 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:21.466085 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:21.466089 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:21.466253 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:21.466570 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:21.466572 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:21.466573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:21.466960 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:21.467100 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:21.467404 (+ 304us) server_negotiation.cc:299] Negotiation successful
0619 14:32:21.467735 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":137,"threads_started":1}
I20260619 14:32:21.468508 6197 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13d21fe2120149a39823873153a21d36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13d21fe2120149a39823873153a21d36" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 42419 } }
I20260619 14:32:21.468838 6197 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13d21fe2120149a39823873153a21d36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:21.468926 6197 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13d21fe2120149a39823873153a21d36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13d21fe2120149a39823873153a21d36, State: Initialized, Role: FOLLOWER
I20260619 14:32:21.469419 6197 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13d21fe2120149a39823873153a21d36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13d21fe2120149a39823873153a21d36" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 42419 } }
I20260619 14:32:21.469573 6197 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13d21fe2120149a39823873153a21d36 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:32:21.469656 6197 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13d21fe2120149a39823873153a21d36 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:32:21.469755 6197 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13d21fe2120149a39823873153a21d36 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:21.470944 6197 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13d21fe2120149a39823873153a21d36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13d21fe2120149a39823873153a21d36" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 42419 } }
I20260619 14:32:21.471287 6197 leader_election.cc:302] T 00000000000000000000000000000000 P 13d21fe2120149a39823873153a21d36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13d21fe2120149a39823873153a21d36; no voters:
I20260619 14:32:21.471609 6197 leader_election.cc:288] T 00000000000000000000000000000000 P 13d21fe2120149a39823873153a21d36 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:32:21.471730 6202 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13d21fe2120149a39823873153a21d36 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:32:21.471958 6202 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13d21fe2120149a39823873153a21d36 [term 1 LEADER]: Becoming Leader. State: Replica: 13d21fe2120149a39823873153a21d36, State: Running, Role: LEADER
I20260619 14:32:21.472291 6202 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13d21fe2120149a39823873153a21d36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13d21fe2120149a39823873153a21d36" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 42419 } }
I20260619 14:32:21.472807 6197 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13d21fe2120149a39823873153a21d36 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:32:21.473909 6204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13d21fe2120149a39823873153a21d36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13d21fe2120149a39823873153a21d36. Latest consensus state: current_term: 1 leader_uuid: "13d21fe2120149a39823873153a21d36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13d21fe2120149a39823873153a21d36" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 42419 } } }
I20260619 14:32:21.474030 6204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13d21fe2120149a39823873153a21d36 [sys.catalog]: This master's current role is: LEADER
I20260619 14:32:21.474435 6211 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:32:21.474391 6203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13d21fe2120149a39823873153a21d36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "13d21fe2120149a39823873153a21d36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13d21fe2120149a39823873153a21d36" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 42419 } } }
I20260619 14:32:21.474646 6203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13d21fe2120149a39823873153a21d36 [sys.catalog]: This master's current role is: LEADER
I20260619 14:32:21.477819 6211 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:32:21.483796 6211 catalog_manager.cc:1347] Generated new cluster ID: 84d04df5875b4eb58daeb0b68c81c370
I20260619 14:32:21.483880 6211 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:32:21.500731 6211 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:32:21.501734 6211 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:32:21.509140 6211 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 13d21fe2120149a39823873153a21d36: Generated new TSK 0
I20260619 14:32:21.509884 6211 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.2.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab.
Entry for principal kudu/127.2.2.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab.
I20260619 14:32:21.547605 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.1:0
--local_ip_for_outbound_sockets=127.2.2.1
--webserver_interface=127.2.2.1
--webserver_port=0
--tserver_master_addrs=127.2.2.62:42419
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab
--principal=kudu/127.2.2.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:44757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:32:21.675527 6223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:21.675824 6223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:21.675870 6223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:21.680081 6223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:21.680153 6223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:21.680219 6223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.1
I20260619 14:32:21.685665 6223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:44757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab
--principal=kudu/127.2.2.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.2.1
--webserver_port=0
--tserver_master_addrs=127.2.2.62:42419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.1
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:32:21.686829 6223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:21.687810 6223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:32:21.697580 6231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:21.697643 6228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:21.697695 6223 server_base.cc:1034] running on GCE node
W20260619 14:32:21.697580 6229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:21.698366 6223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:21.699105 6223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:21.700304 6223 hybrid_clock.cc:648] HybridClock initialized: now 1781879541700280 us; error 38 us; skew 500 ppm
Jun 19 14:32:21 dist-test-slave-70rm krb5kdc[6117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879541, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:32:21.703289 6223 init.cc:377] Logged in from keytab as kudu/127.2.2.1@KRBTEST.COM (short username kudu)
I20260619 14:32:21.704005 6223 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:21.704069 6223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:21.704234 6223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:32:21.706442 6223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/ts-0/data/instance:
uuid: "7de9641318e646c2a322d87765a4fad4"
format_stamp: "Formatted at 2026-06-19 14:32:21 on dist-test-slave-70rm"
server_key: "02782db3bd317a25e500c216e0279891"
server_key_iv: "38e0a4c040627cc8a120faacc14ecbf5"
server_key_version: "encryptionkey@0"
I20260619 14:32:21.707015 6223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/ts-0/wal/instance:
uuid: "7de9641318e646c2a322d87765a4fad4"
format_stamp: "Formatted at 2026-06-19 14:32:21 on dist-test-slave-70rm"
server_key: "02782db3bd317a25e500c216e0279891"
server_key_iv: "38e0a4c040627cc8a120faacc14ecbf5"
server_key_version: "encryptionkey@0"
I20260619 14:32:21.710861 6223 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260619 14:32:21.713361 6237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:21.714737 6223 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:32:21.714828 6223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/ts-0/wal
uuid: "7de9641318e646c2a322d87765a4fad4"
format_stamp: "Formatted at 2026-06-19 14:32:21 on dist-test-slave-70rm"
server_key: "02782db3bd317a25e500c216e0279891"
server_key_iv: "38e0a4c040627cc8a120faacc14ecbf5"
server_key_version: "encryptionkey@0"
I20260619 14:32:21.714915 6223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:21.729220 6223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:21.732143 6223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:32:21.732296 6223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:21.732821 6223 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:32:21.733603 6223 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:32:21.733649 6223 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:21.733688 6223 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:32:21.733703 6223 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:21.742509 6223 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.1:37947
I20260619 14:32:21.742527 6350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.1:37947 every 8 connection(s)
I20260619 14:32:21.743144 6223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
I20260619 14:32:21.745247 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 6223
I20260619 14:32:21.745358 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/ts-0/wal/instance
I20260619 14:32:21.745637 2056 external_mini_cluster.cc:1428] Setting key 28520799971b500fcf2ae83cca0db2bb
Jun 19 14:32:21 dist-test-slave-70rm krb5kdc[6117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879541, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.1@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.2.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.2@KRBTEST.COM" created.
I20260619 14:32:21.757534 6200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:21.744875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:57777 (local address 127.2.2.62:42419)
0619 14:32:21.745024 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:21.745029 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:21.746137 (+ 1108us) server_negotiation.cc:407] Connection header received
0619 14:32:21.747558 (+ 1421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:21.747561 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:21.747604 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:21.747673 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:21.749227 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:21.749772 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:21.751063 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:21.751300 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:21.751410 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:21.753867 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:21.753871 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:21.753873 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:21.753897 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:21.755410 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:21.755899 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:21.755902 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:21.755903 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:21.755976 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:21.756299 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:21.756305 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:21.756310 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:21.756494 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:21.756569 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:21.757196 (+ 627us) server_negotiation.cc:299] Negotiation successful
0619 14:32:21.757348 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 14:32:21.758378 6353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:21.745336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:42419 (local address 127.2.2.1:57777)
0619 14:32:21.745830 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:21.745884 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:21.747323 (+ 1439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:21.747869 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:21.747881 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:21.748435 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:21.749075 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:21.749090 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:21.749900 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:21.749904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:21.750948 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:21.750956 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:21.751395 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:21.751450 (+ 55us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:21.751620 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:21.752228 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:21.752245 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:21.753643 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:21.755534 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:21.755539 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:21.755554 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:21.755808 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:21.756060 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:21.756062 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:21.756063 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:21.756192 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:21.756580 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:21.756585 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:21.756886 (+ 301us) client_negotiation.cc:769] Sending connection context
0619 14:32:21.757139 (+ 253us) client_negotiation.cc:240] Negotiation successful
0619 14:32:21.757446 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":131,"threads_started":1}
I20260619 14:32:21.759688 6351 heartbeater.cc:347] Connected to a master server at 127.2.2.62:42419
I20260619 14:32:21.759931 6351 heartbeater.cc:464] Registering TS with master...
I20260619 14:32:21.760392 6351 heartbeater.cc:511] Master 127.2.2.62:42419 requested a full tablet report, sending...
I20260619 14:32:21.761878 6161 ts_manager.cc:194] Registered new tserver with Master: 7de9641318e646c2a322d87765a4fad4 (127.2.2.1:37947)
I20260619 14:32:21.763041 6161 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.2.1@KRBTEST.COM'} at 127.2.2.1:57777
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab.
Entry for principal kudu/127.2.2.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab.
I20260619 14:32:21.773178 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.2:0
--local_ip_for_outbound_sockets=127.2.2.2
--webserver_interface=127.2.2.2
--webserver_port=0
--tserver_master_addrs=127.2.2.62:42419
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab
--principal=kudu/127.2.2.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:44757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:32:21.886466 6356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:21.886809 6356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:21.886911 6356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:21.891824 6356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:21.891933 6356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:21.892040 6356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.2
I20260619 14:32:21.897897 6356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:44757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab
--principal=kudu/127.2.2.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.2.2
--webserver_port=0
--tserver_master_addrs=127.2.2.62:42419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.2
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:32:21.899132 6356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:21.900277 6356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:32:21.909909 6364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:21.909909 6361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:21.909968 6362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:21.910405 6356 server_base.cc:1034] running on GCE node
I20260619 14:32:21.911640 6356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:21.912379 6356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:21.913573 6356 hybrid_clock.cc:648] HybridClock initialized: now 1781879541913548 us; error 40 us; skew 500 ppm
Jun 19 14:32:21 dist-test-slave-70rm krb5kdc[6117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879541, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:32:21.916512 6356 init.cc:377] Logged in from keytab as kudu/127.2.2.2@KRBTEST.COM (short username kudu)
I20260619 14:32:21.917212 6356 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:21.917271 6356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:21.917436 6356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:32:21.919737 6356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/ts-1/data/instance:
uuid: "30e30f237517411ea5e54c172fb7c268"
format_stamp: "Formatted at 2026-06-19 14:32:21 on dist-test-slave-70rm"
server_key: "9eb5ecf40748cc132c7ac62798a43505"
server_key_iv: "874f1b44f3ad6caea7b7e849703b239b"
server_key_version: "encryptionkey@0"
I20260619 14:32:21.920255 6356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/ts-1/wal/instance:
uuid: "30e30f237517411ea5e54c172fb7c268"
format_stamp: "Formatted at 2026-06-19 14:32:21 on dist-test-slave-70rm"
server_key: "9eb5ecf40748cc132c7ac62798a43505"
server_key_iv: "874f1b44f3ad6caea7b7e849703b239b"
server_key_version: "encryptionkey@0"
I20260619 14:32:21.924154 6356 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:32:21.926723 6370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:21.928049 6356 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:32:21.928175 6356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/ts-1/wal
uuid: "30e30f237517411ea5e54c172fb7c268"
format_stamp: "Formatted at 2026-06-19 14:32:21 on dist-test-slave-70rm"
server_key: "9eb5ecf40748cc132c7ac62798a43505"
server_key_iv: "874f1b44f3ad6caea7b7e849703b239b"
server_key_version: "encryptionkey@0"
I20260619 14:32:21.928296 6356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:21.942929 6356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:21.946547 6356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:32:21.946762 6356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:21.947365 6356 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:32:21.948230 6356 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:32:21.948277 6356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:21.948316 6356 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:32:21.948331 6356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:21.957221 6356 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.2:37927
I20260619 14:32:21.957244 6483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.2:37927 every 8 connection(s)
I20260619 14:32:21.957834 6356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
I20260619 14:32:21.959990 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 6356
I20260619 14:32:21.960090 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/ts-1/wal/instance
I20260619 14:32:21.960441 2056 external_mini_cluster.cc:1428] Setting key b49fc6de2d62e6390650ec0db28e1f2f
Jun 19 14:32:21 dist-test-slave-70rm krb5kdc[6117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879541, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.2@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.2.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.3@KRBTEST.COM" created.
I20260619 14:32:21.974195 6200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:21.959632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:45931 (local address 127.2.2.62:42419)
0619 14:32:21.959808 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:21.959814 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:21.960814 (+ 1000us) server_negotiation.cc:407] Connection header received
0619 14:32:21.962539 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:21.962545 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:21.962605 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:21.962690 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:21.964179 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:21.964707 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:21.966560 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:21.966785 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:21.966864 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:21.969818 (+ 2954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:21.969821 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:21.969824 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:21.969850 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:21.971625 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:21.972240 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:21.972245 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:21.972252 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:21.972331 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:21.972703 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:21.972708 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:21.972710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:21.972968 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:21.973086 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:21.973669 (+ 583us) server_negotiation.cc:299] Negotiation successful
0619 14:32:21.973874 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 14:32:21.975101 6486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:21.960055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:42419 (local address 127.2.2.2:45931)
0619 14:32:21.960626 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:21.960685 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:21.962304 (+ 1619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:21.962838 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:21.962844 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:21.963268 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:21.964019 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:21.964039 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:21.964838 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:21.964844 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:21.966407 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:21.966419 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:21.966886 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:21.966949 (+ 63us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:21.967190 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:21.967880 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:21.967903 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:21.969621 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:21.971755 (+ 2134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:21.971761 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:21.971780 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:21.972090 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:21.972425 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:21.972430 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:21.972432 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:21.972585 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:21.973084 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:21.973091 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:21.973348 (+ 257us) client_negotiation.cc:769] Sending connection context
0619 14:32:21.973588 (+ 240us) client_negotiation.cc:240] Negotiation successful
0619 14:32:21.974025 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":111,"threads_started":1}
I20260619 14:32:21.976311 6484 heartbeater.cc:347] Connected to a master server at 127.2.2.62:42419
I20260619 14:32:21.976557 6484 heartbeater.cc:464] Registering TS with master...
I20260619 14:32:21.977079 6484 heartbeater.cc:511] Master 127.2.2.62:42419 requested a full tablet report, sending...
I20260619 14:32:21.978199 6161 ts_manager.cc:194] Registered new tserver with Master: 30e30f237517411ea5e54c172fb7c268 (127.2.2.2:37927)
I20260619 14:32:21.978835 6161 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.2.2@KRBTEST.COM'} at 127.2.2.2:45931
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab.
Entry for principal kudu/127.2.2.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab.
I20260619 14:32:21.988992 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.3:0
--local_ip_for_outbound_sockets=127.2.2.3
--webserver_interface=127.2.2.3
--webserver_port=0
--tserver_master_addrs=127.2.2.62:42419
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab
--principal=kudu/127.2.2.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:44757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:32:22.097900 6489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:22.098214 6489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:22.098315 6489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:22.102451 6489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:22.102526 6489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:22.102632 6489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.3
I20260619 14:32:22.108456 6489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:44757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab
--principal=kudu/127.2.2.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.2.3
--webserver_port=0
--tserver_master_addrs=127.2.2.62:42419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.3
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:32:22.109628 6489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:22.110908 6489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:32:22.121827 6495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:22.122082 6489 server_base.cc:1034] running on GCE node
W20260619 14:32:22.121827 6497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:22.121827 6494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:22.122677 6489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:22.123422 6489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:22.124627 6489 hybrid_clock.cc:648] HybridClock initialized: now 1781879542124610 us; error 39 us; skew 500 ppm
Jun 19 14:32:22 dist-test-slave-70rm krb5kdc[6117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879542, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:32:22.127746 6489 init.cc:377] Logged in from keytab as kudu/127.2.2.3@KRBTEST.COM (short username kudu)
I20260619 14:32:22.128582 6489 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:22.128679 6489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:22.128926 6489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:32:22.131223 6489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/ts-2/data/instance:
uuid: "587e8b5a872f472c91f40242e1c13948"
format_stamp: "Formatted at 2026-06-19 14:32:22 on dist-test-slave-70rm"
server_key: "4beb2fb7e54b12827c8bd6567e5e3f22"
server_key_iv: "35244aaec5837ab56555fc49ce86d5b4"
server_key_version: "encryptionkey@0"
I20260619 14:32:22.131819 6489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/ts-2/wal/instance:
uuid: "587e8b5a872f472c91f40242e1c13948"
format_stamp: "Formatted at 2026-06-19 14:32:22 on dist-test-slave-70rm"
server_key: "4beb2fb7e54b12827c8bd6567e5e3f22"
server_key_iv: "35244aaec5837ab56555fc49ce86d5b4"
server_key_version: "encryptionkey@0"
I20260619 14:32:22.136040 6489 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:32:22.138625 6503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:22.139933 6489 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:32:22.140112 6489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/ts-2/wal
uuid: "587e8b5a872f472c91f40242e1c13948"
format_stamp: "Formatted at 2026-06-19 14:32:22 on dist-test-slave-70rm"
server_key: "4beb2fb7e54b12827c8bd6567e5e3f22"
server_key_iv: "35244aaec5837ab56555fc49ce86d5b4"
server_key_version: "encryptionkey@0"
I20260619 14:32:22.140252 6489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:22.164579 6489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:22.168154 6489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:32:22.168378 6489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:22.168996 6489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:32:22.169855 6489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:32:22.169924 6489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:22.169996 6489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:32:22.170038 6489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:22.179891 6489 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.3:45747
I20260619 14:32:22.179908 6616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.3:45747 every 8 connection(s)
I20260619 14:32:22.180536 6489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
I20260619 14:32:22.185730 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 6489
I20260619 14:32:22.185868 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/ts-2/wal/instance
I20260619 14:32:22.186235 2056 external_mini_cluster.cc:1428] Setting key 61c1059dcf6138a856a1fc7c54741508
Jun 19 14:32:22 dist-test-slave-70rm krb5kdc[6117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879542, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.3@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:32:22.195228 6200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:22.182484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.3:53079 (local address 127.2.2.62:42419)
0619 14:32:22.182629 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:22.182632 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:22.183480 (+ 848us) server_negotiation.cc:407] Connection header received
0619 14:32:22.184874 (+ 1394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:22.184877 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:22.184934 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:22.185107 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:22.186719 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:22.187353 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:22.188501 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:22.188736 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:22.188844 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:22.191285 (+ 2441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:22.191289 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:22.191292 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:22.191326 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:22.192970 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:22.193548 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:22.193552 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:22.193554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:22.193640 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:22.194013 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:22.194016 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:22.194018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:22.194281 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:22.194392 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:22.194922 (+ 530us) server_negotiation.cc:299] Negotiation successful
0619 14:32:22.195053 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 14:32:22.196103 6619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:22.182887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:42419 (local address 127.2.2.3:53079)
0619 14:32:22.183333 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:22.183377 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:22.184654 (+ 1277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:22.185285 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:22.185293 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:22.185836 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:22.186570 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:22.186587 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:22.187476 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:22.187479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:22.188377 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:22.188386 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:22.188830 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:22.188887 (+ 57us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:22.189064 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:22.189613 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:22.189630 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:22.191113 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:22.193120 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:22.193127 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:22.193145 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:22.193433 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:22.193743 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:22.193746 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:22.193748 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:22.193907 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:22.194407 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:22.194415 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:22.194674 (+ 259us) client_negotiation.cc:769] Sending connection context
0619 14:32:22.194882 (+ 208us) client_negotiation.cc:240] Negotiation successful
0619 14:32:22.195227 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":110,"threads_started":1}
I20260619 14:32:22.197293 6617 heartbeater.cc:347] Connected to a master server at 127.2.2.62:42419
I20260619 14:32:22.197520 6617 heartbeater.cc:464] Registering TS with master...
I20260619 14:32:22.197942 6617 heartbeater.cc:511] Master 127.2.2.62:42419 requested a full tablet report, sending...
I20260619 14:32:22.199054 6161 ts_manager.cc:194] Registered new tserver with Master: 587e8b5a872f472c91f40242e1c13948 (127.2.2.3:45747)
I20260619 14:32:22.199801 6161 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.2.3@KRBTEST.COM'} at 127.2.2.3:53079
I20260619 14:32:22.199925 2056 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:32:22.208654 6200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:22.202024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55610 (local address 127.2.2.62:42419)
0619 14:32:22.202175 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:22.202180 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:22.202357 (+ 177us) server_negotiation.cc:407] Connection header received
0619 14:32:22.202443 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:22.202445 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:22.202481 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:22.202543 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:22.203140 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:22.203636 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:22.204428 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:22.204629 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:22.204701 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:22.205910 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:22.205913 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:22.205915 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:22.205936 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:22.207172 (+ 1236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:22.207568 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:22.207570 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:22.207571 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:22.207645 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:22.207861 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:22.207863 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:22.207864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:22.208060 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:22.208160 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:22.208362 (+ 202us) server_negotiation.cc:299] Negotiation successful
0619 14:32:22.208453 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 14:32:22.211491 6161 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55610:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:32:22.214603 6161 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:32:22.228698 6628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:22.223153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.1:37947 (local address 127.0.0.1:42754)
0619 14:32:22.223536 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:22.223589 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:22.223720 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:22.224487 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:22.224492 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:22.224517 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:32:22.224616 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:22.224621 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:22.225937 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:22.225944 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:22.226944 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:22.226953 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:22.228071 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:22.228074 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:22.228189 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:22.228227 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 14:32:22.228400 (+ 173us) client_negotiation.cc:240] Negotiation successful
0619 14:32:22.228538 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":105,"threads_started":1}
I20260619 14:32:22.228704 6629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:22.223347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42754 (local address 127.2.2.1:37947)
0619 14:32:22.223632 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:22.223642 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:22.223667 (+ 25us) server_negotiation.cc:407] Connection header received
0619 14:32:22.223893 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:22.223899 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:22.224130 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:22.224297 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:32:22.224736 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:22.225814 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:22.227120 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:22.227940 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:22.228073 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:22.228228 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:22.228442 (+ 214us) server_negotiation.cc:299] Negotiation successful
0619 14:32:22.228549 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":79,"threads_started":1}
I20260619 14:32:22.229858 6630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:22.223916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.3:45747 (local address 127.0.0.1:38504)
0619 14:32:22.224141 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:22.224154 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:22.224247 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:22.225533 (+ 1286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:22.225536 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:22.225555 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:32:22.225655 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:22.225661 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:22.227113 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:22.227116 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:22.228338 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:22.228348 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:22.229449 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:22.229453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:22.229545 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:22.229571 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 14:32:22.229661 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 14:32:22.229732 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":102,"threads_started":1}
I20260619 14:32:22.229972 6631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:22.224071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38504 (local address 127.2.2.3:45747)
0619 14:32:22.225056 (+ 985us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:22.225064 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:22.225082 (+ 18us) server_negotiation.cc:407] Connection header received
0619 14:32:22.225179 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:22.225183 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:22.225371 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:22.225527 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:32:22.225768 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:22.226965 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:22.228489 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:22.229316 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:22.229424 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:22.229551 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:22.229697 (+ 146us) server_negotiation.cc:299] Negotiation successful
0619 14:32:22.229790 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":892,"thread_start_us":150,"threads_started":1}
I20260619 14:32:22.230383 6632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:22.224374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.2:37927 (local address 127.0.0.1:42322)
0619 14:32:22.224594 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:22.224607 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:22.224681 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:22.225694 (+ 1013us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:22.225697 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:22.225717 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:32:22.225830 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:22.225836 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:22.227393 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:22.227396 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:22.228488 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:22.228495 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:22.229922 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:22.229925 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:22.230029 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:22.230059 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 14:32:22.230111 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 14:32:22.230268 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":96,"threads_started":1}
I20260619 14:32:22.231211 6633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:22.224977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42322 (local address 127.2.2.2:37927)
0619 14:32:22.225271 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:22.225282 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:22.225301 (+ 19us) server_negotiation.cc:407] Connection header received
0619 14:32:22.225367 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:22.225372 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:22.225549 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:22.225974 (+ 425us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:32:22.226153 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:22.227258 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:22.228615 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:22.229790 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:22.230755 (+ 965us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:22.230913 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:22.230966 (+ 53us) server_negotiation.cc:299] Negotiation successful
0619 14:32:22.231057 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":90,"threads_started":1}
I20260619 14:32:22.232052 6285 tablet_service.cc:1467] Processing CreateTablet for tablet 66bf7ad027c14aae8349191197dc6958 (DEFAULT_TABLE table=test-table [id=02ad1a87b2fa46e2a9008505bf3170b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:32:22.232667 6551 tablet_service.cc:1467] Processing CreateTablet for tablet 66bf7ad027c14aae8349191197dc6958 (DEFAULT_TABLE table=test-table [id=02ad1a87b2fa46e2a9008505bf3170b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:32:22.233119 6285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66bf7ad027c14aae8349191197dc6958. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:22.233541 6551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66bf7ad027c14aae8349191197dc6958. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:22.239454 6634 tablet_bootstrap.cc:492] T 66bf7ad027c14aae8349191197dc6958 P 587e8b5a872f472c91f40242e1c13948: Bootstrap starting.
I20260619 14:32:22.241089 6418 tablet_service.cc:1467] Processing CreateTablet for tablet 66bf7ad027c14aae8349191197dc6958 (DEFAULT_TABLE table=test-table [id=02ad1a87b2fa46e2a9008505bf3170b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:32:22.241578 6634 tablet_bootstrap.cc:654] T 66bf7ad027c14aae8349191197dc6958 P 587e8b5a872f472c91f40242e1c13948: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:22.241988 6418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66bf7ad027c14aae8349191197dc6958. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:22.242733 6634 log.cc:824] T 66bf7ad027c14aae8349191197dc6958 P 587e8b5a872f472c91f40242e1c13948: Log is configured to *not* fsync() on all Append() calls
I20260619 14:32:22.243739 6635 tablet_bootstrap.cc:492] T 66bf7ad027c14aae8349191197dc6958 P 7de9641318e646c2a322d87765a4fad4: Bootstrap starting.
I20260619 14:32:22.245471 6635 tablet_bootstrap.cc:654] T 66bf7ad027c14aae8349191197dc6958 P 7de9641318e646c2a322d87765a4fad4: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:22.246562 6635 log.cc:824] T 66bf7ad027c14aae8349191197dc6958 P 7de9641318e646c2a322d87765a4fad4: Log is configured to *not* fsync() on all Append() calls
I20260619 14:32:22.248510 6635 tablet_bootstrap.cc:492] T 66bf7ad027c14aae8349191197dc6958 P 7de9641318e646c2a322d87765a4fad4: No bootstrap required, opened a new log
I20260619 14:32:22.248512 6634 tablet_bootstrap.cc:492] T 66bf7ad027c14aae8349191197dc6958 P 587e8b5a872f472c91f40242e1c13948: No bootstrap required, opened a new log
I20260619 14:32:22.248721 6635 ts_tablet_manager.cc:1397] T 66bf7ad027c14aae8349191197dc6958 P 7de9641318e646c2a322d87765a4fad4: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:32:22.248721 6634 ts_tablet_manager.cc:1397] T 66bf7ad027c14aae8349191197dc6958 P 587e8b5a872f472c91f40242e1c13948: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260619 14:32:22.249230 6638 tablet_bootstrap.cc:492] T 66bf7ad027c14aae8349191197dc6958 P 30e30f237517411ea5e54c172fb7c268: Bootstrap starting.
I20260619 14:32:22.251345 6638 tablet_bootstrap.cc:654] T 66bf7ad027c14aae8349191197dc6958 P 30e30f237517411ea5e54c172fb7c268: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:22.252408 6638 log.cc:824] T 66bf7ad027c14aae8349191197dc6958 P 30e30f237517411ea5e54c172fb7c268: Log is configured to *not* fsync() on all Append() calls
I20260619 14:32:22.252727 6635 raft_consensus.cc:348] T 66bf7ad027c14aae8349191197dc6958 P 7de9641318e646c2a322d87765a4fad4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7de9641318e646c2a322d87765a4fad4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37947 } } peers { permanent_uuid: "587e8b5a872f472c91f40242e1c13948" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45747 } } peers { permanent_uuid: "30e30f237517411ea5e54c172fb7c268" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 37927 } }
I20260619 14:32:22.252735 6634 raft_consensus.cc:348] T 66bf7ad027c14aae8349191197dc6958 P 587e8b5a872f472c91f40242e1c13948 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7de9641318e646c2a322d87765a4fad4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37947 } } peers { permanent_uuid: "587e8b5a872f472c91f40242e1c13948" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45747 } } peers { permanent_uuid: "30e30f237517411ea5e54c172fb7c268" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 37927 } }
I20260619 14:32:22.253008 6634 raft_consensus.cc:374] T 66bf7ad027c14aae8349191197dc6958 P 587e8b5a872f472c91f40242e1c13948 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:22.253008 6635 raft_consensus.cc:374] T 66bf7ad027c14aae8349191197dc6958 P 7de9641318e646c2a322d87765a4fad4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:22.253067 6635 raft_consensus.cc:729] T 66bf7ad027c14aae8349191197dc6958 P 7de9641318e646c2a322d87765a4fad4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7de9641318e646c2a322d87765a4fad4, State: Initialized, Role: FOLLOWER
I20260619 14:32:22.253067 6634 raft_consensus.cc:729] T 66bf7ad027c14aae8349191197dc6958 P 587e8b5a872f472c91f40242e1c13948 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 587e8b5a872f472c91f40242e1c13948, State: Initialized, Role: FOLLOWER
I20260619 14:32:22.253537 6634 consensus_queue.cc:260] T 66bf7ad027c14aae8349191197dc6958 P 587e8b5a872f472c91f40242e1c13948 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7de9641318e646c2a322d87765a4fad4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37947 } } peers { permanent_uuid: "587e8b5a872f472c91f40242e1c13948" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45747 } } peers { permanent_uuid: "30e30f237517411ea5e54c172fb7c268" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 37927 } }
I20260619 14:32:22.253537 6635 consensus_queue.cc:260] T 66bf7ad027c14aae8349191197dc6958 P 7de9641318e646c2a322d87765a4fad4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7de9641318e646c2a322d87765a4fad4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37947 } } peers { permanent_uuid: "587e8b5a872f472c91f40242e1c13948" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45747 } } peers { permanent_uuid: "30e30f237517411ea5e54c172fb7c268" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 37927 } }
I20260619 14:32:22.254240 6617 heartbeater.cc:503] Master 127.2.2.62:42419 was elected leader, sending a full tablet report...
I20260619 14:32:22.254511 6351 heartbeater.cc:503] Master 127.2.2.62:42419 was elected leader, sending a full tablet report...
I20260619 14:32:22.254622 6638 tablet_bootstrap.cc:492] T 66bf7ad027c14aae8349191197dc6958 P 30e30f237517411ea5e54c172fb7c268: No bootstrap required, opened a new log
I20260619 14:32:22.254948 6638 ts_tablet_manager.cc:1397] T 66bf7ad027c14aae8349191197dc6958 P 30e30f237517411ea5e54c172fb7c268: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.004s
I20260619 14:32:22.255224 6634 ts_tablet_manager.cc:1428] T 66bf7ad027c14aae8349191197dc6958 P 587e8b5a872f472c91f40242e1c13948: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 14:32:22.257437 6635 ts_tablet_manager.cc:1428] T 66bf7ad027c14aae8349191197dc6958 P 7de9641318e646c2a322d87765a4fad4: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260619 14:32:22.259232 6638 raft_consensus.cc:348] T 66bf7ad027c14aae8349191197dc6958 P 30e30f237517411ea5e54c172fb7c268 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7de9641318e646c2a322d87765a4fad4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37947 } } peers { permanent_uuid: "587e8b5a872f472c91f40242e1c13948" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45747 } } peers { permanent_uuid: "30e30f237517411ea5e54c172fb7c268" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 37927 } }
I20260619 14:32:22.259544 6638 raft_consensus.cc:374] T 66bf7ad027c14aae8349191197dc6958 P 30e30f237517411ea5e54c172fb7c268 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:22.259604 6638 raft_consensus.cc:729] T 66bf7ad027c14aae8349191197dc6958 P 30e30f237517411ea5e54c172fb7c268 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30e30f237517411ea5e54c172fb7c268, State: Initialized, Role: FOLLOWER
I20260619 14:32:22.260043 6638 consensus_queue.cc:260] T 66bf7ad027c14aae8349191197dc6958 P 30e30f237517411ea5e54c172fb7c268 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7de9641318e646c2a322d87765a4fad4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37947 } } peers { permanent_uuid: "587e8b5a872f472c91f40242e1c13948" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45747 } } peers { permanent_uuid: "30e30f237517411ea5e54c172fb7c268" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 37927 } }
I20260619 14:32:22.260687 6484 heartbeater.cc:503] Master 127.2.2.62:42419 was elected leader, sending a full tablet report...
I20260619 14:32:22.260978 6638 ts_tablet_manager.cc:1428] T 66bf7ad027c14aae8349191197dc6958 P 30e30f237517411ea5e54c172fb7c268: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 14:32:22.313885 6641 raft_consensus.cc:482] T 66bf7ad027c14aae8349191197dc6958 P 587e8b5a872f472c91f40242e1c13948 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:32:22.314111 6641 raft_consensus.cc:504] T 66bf7ad027c14aae8349191197dc6958 P 587e8b5a872f472c91f40242e1c13948 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7de9641318e646c2a322d87765a4fad4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37947 } } peers { permanent_uuid: "587e8b5a872f472c91f40242e1c13948" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45747 } } peers { permanent_uuid: "30e30f237517411ea5e54c172fb7c268" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 37927 } }
I20260619 14:32:22.315277 6641 leader_election.cc:288] T 66bf7ad027c14aae8349191197dc6958 P 587e8b5a872f472c91f40242e1c13948 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7de9641318e646c2a322d87765a4fad4 (127.2.2.1:37947), 30e30f237517411ea5e54c172fb7c268 (127.2.2.2:37927)
I20260619 14:32:22.319540 6619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:22.315540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.1:37947 (local address 127.2.2.3:38597)
0619 14:32:22.315676 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:22.315692 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:22.315795 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:22.316093 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:22.316096 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:22.316122 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:32:22.316273 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:22.316278 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:22.317312 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:22.317315 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:22.318297 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:22.318306 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:22.319035 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:22.319038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:22.319175 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:22.319215 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 14:32:22.319284 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 14:32:22.319359 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":32,"spinlock_wait_cycles":384}
I20260619 14:32:22.319564 6629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:22.315730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.3:38597 (local address 127.2.2.1:37947)
0619 14:32:22.315857 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:22.315861 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:22.315876 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:32:22.315934 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:22.315937 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:22.315987 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:22.316055 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:32:22.316412 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:22.317202 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:22.318393 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:22.318884 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:22.319028 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:22.319101 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:22.319311 (+ 210us) server_negotiation.cc:299] Negotiation successful
0619 14:32:22.319410 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 14:32:22.319730 6643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:22.315543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.2:37927 (local address 127.2.2.3:35333)
0619 14:32:22.316011 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:22.316022 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:22.316131 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:22.316348 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:22.316354 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:22.316375 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:32:22.316505 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:22.316509 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:22.317440 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:22.317445 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:22.318499 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:22.318517 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:22.319240 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:22.319243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:22.319371 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:22.319394 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 14:32:22.319504 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 14:32:22.319578 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"mutex_wait_us":137,"spinlock_wait_cycles":3328,"thread_start_us":133,"threads_started":1}
I20260619 14:32:22.319773 6633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:22.315645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.3:35333 (local address 127.2.2.2:37927)
0619 14:32:22.315772 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:22.315776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:22.316131 (+ 355us) server_negotiation.cc:407] Connection header received
0619 14:32:22.316212 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:22.316214 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:22.316256 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:22.316315 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:32:22.316610 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:22.317335 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:22.318616 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:22.319125 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:22.319228 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:22.319303 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:22.319541 (+ 238us) server_negotiation.cc:299] Negotiation successful
0619 14:32:22.319630 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 14:32:22.320304 6305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66bf7ad027c14aae8349191197dc6958" candidate_uuid: "587e8b5a872f472c91f40242e1c13948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7de9641318e646c2a322d87765a4fad4" is_pre_election: true
I20260619 14:32:22.320303 6438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66bf7ad027c14aae8349191197dc6958" candidate_uuid: "587e8b5a872f472c91f40242e1c13948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30e30f237517411ea5e54c172fb7c268" is_pre_election: true
I20260619 14:32:22.320621 6305 raft_consensus.cc:2454] T 66bf7ad027c14aae8349191197dc6958 P 7de9641318e646c2a322d87765a4fad4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 587e8b5a872f472c91f40242e1c13948 in term 0.
I20260619 14:32:22.320621 6438 raft_consensus.cc:2454] T 66bf7ad027c14aae8349191197dc6958 P 30e30f237517411ea5e54c172fb7c268 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 587e8b5a872f472c91f40242e1c13948 in term 0.
I20260619 14:32:22.321174 6507 leader_election.cc:302] T 66bf7ad027c14aae8349191197dc6958 P 587e8b5a872f472c91f40242e1c13948 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 30e30f237517411ea5e54c172fb7c268, 587e8b5a872f472c91f40242e1c13948, 7de9641318e646c2a322d87765a4fad4; no voters:
I20260619 14:32:22.321456 6641 raft_consensus.cc:2781] T 66bf7ad027c14aae8349191197dc6958 P 587e8b5a872f472c91f40242e1c13948 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:32:22.321538 6641 raft_consensus.cc:482] T 66bf7ad027c14aae8349191197dc6958 P 587e8b5a872f472c91f40242e1c13948 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:32:22.321583 6641 raft_consensus.cc:3037] T 66bf7ad027c14aae8349191197dc6958 P 587e8b5a872f472c91f40242e1c13948 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:22.322665 6641 raft_consensus.cc:504] T 66bf7ad027c14aae8349191197dc6958 P 587e8b5a872f472c91f40242e1c13948 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7de9641318e646c2a322d87765a4fad4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37947 } } peers { permanent_uuid: "587e8b5a872f472c91f40242e1c13948" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45747 } } peers { permanent_uuid: "30e30f237517411ea5e54c172fb7c268" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 37927 } }
I20260619 14:32:22.323005 6641 leader_election.cc:288] T 66bf7ad027c14aae8349191197dc6958 P 587e8b5a872f472c91f40242e1c13948 [CANDIDATE]: Term 1 election: Requested vote from peers 7de9641318e646c2a322d87765a4fad4 (127.2.2.1:37947), 30e30f237517411ea5e54c172fb7c268 (127.2.2.2:37927)
I20260619 14:32:22.323386 6305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66bf7ad027c14aae8349191197dc6958" candidate_uuid: "587e8b5a872f472c91f40242e1c13948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7de9641318e646c2a322d87765a4fad4"
I20260619 14:32:22.323386 6438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66bf7ad027c14aae8349191197dc6958" candidate_uuid: "587e8b5a872f472c91f40242e1c13948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30e30f237517411ea5e54c172fb7c268"
I20260619 14:32:22.323513 6305 raft_consensus.cc:3037] T 66bf7ad027c14aae8349191197dc6958 P 7de9641318e646c2a322d87765a4fad4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:22.323523 6438 raft_consensus.cc:3037] T 66bf7ad027c14aae8349191197dc6958 P 30e30f237517411ea5e54c172fb7c268 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:22.324770 6305 raft_consensus.cc:2454] T 66bf7ad027c14aae8349191197dc6958 P 7de9641318e646c2a322d87765a4fad4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 587e8b5a872f472c91f40242e1c13948 in term 1.
I20260619 14:32:22.324770 6438 raft_consensus.cc:2454] T 66bf7ad027c14aae8349191197dc6958 P 30e30f237517411ea5e54c172fb7c268 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 587e8b5a872f472c91f40242e1c13948 in term 1.
I20260619 14:32:22.325073 6507 leader_election.cc:302] T 66bf7ad027c14aae8349191197dc6958 P 587e8b5a872f472c91f40242e1c13948 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30e30f237517411ea5e54c172fb7c268, 587e8b5a872f472c91f40242e1c13948; no voters:
I20260619 14:32:22.325264 6641 raft_consensus.cc:2781] T 66bf7ad027c14aae8349191197dc6958 P 587e8b5a872f472c91f40242e1c13948 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:32:22.325522 6641 raft_consensus.cc:686] T 66bf7ad027c14aae8349191197dc6958 P 587e8b5a872f472c91f40242e1c13948 [term 1 LEADER]: Becoming Leader. State: Replica: 587e8b5a872f472c91f40242e1c13948, State: Running, Role: LEADER
I20260619 14:32:22.325852 6641 consensus_queue.cc:237] T 66bf7ad027c14aae8349191197dc6958 P 587e8b5a872f472c91f40242e1c13948 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7de9641318e646c2a322d87765a4fad4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37947 } } peers { permanent_uuid: "587e8b5a872f472c91f40242e1c13948" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45747 } } peers { permanent_uuid: "30e30f237517411ea5e54c172fb7c268" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 37927 } }
I20260619 14:32:22.329160 6160 catalog_manager.cc:5495] T 66bf7ad027c14aae8349191197dc6958 P 587e8b5a872f472c91f40242e1c13948 reported cstate change: term changed from 0 to 1, leader changed from <none> to 587e8b5a872f472c91f40242e1c13948 (127.2.2.3). New cstate: current_term: 1 leader_uuid: "587e8b5a872f472c91f40242e1c13948" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7de9641318e646c2a322d87765a4fad4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "587e8b5a872f472c91f40242e1c13948" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30e30f237517411ea5e54c172fb7c268" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 37927 } health_report { overall_health: UNKNOWN } } }
I20260619 14:32:22.355872 6631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:22.352524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38506 (local address 127.2.2.3:45747)
0619 14:32:22.352633 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:22.352636 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:22.352699 (+ 63us) server_negotiation.cc:407] Connection header received
0619 14:32:22.352828 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:22.352830 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:22.352884 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:22.352970 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:32:22.353198 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:22.353705 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:22.354536 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:22.354745 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:22.354822 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:22.355145 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:32:22.355570 (+ 425us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:32:22.355658 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:22.355716 (+ 58us) server_negotiation.cc:299] Negotiation successful
0619 14:32:22.355763 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 14:32:22.361901 6438 raft_consensus.cc:1261] T 66bf7ad027c14aae8349191197dc6958 P 30e30f237517411ea5e54c172fb7c268 [term 1 FOLLOWER]: Refusing update from remote peer 587e8b5a872f472c91f40242e1c13948: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:32:22.361902 6305 raft_consensus.cc:1261] T 66bf7ad027c14aae8349191197dc6958 P 7de9641318e646c2a322d87765a4fad4 [term 1 FOLLOWER]: Refusing update from remote peer 587e8b5a872f472c91f40242e1c13948: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:32:22.362695 6644 consensus_queue.cc:1035] T 66bf7ad027c14aae8349191197dc6958 P 587e8b5a872f472c91f40242e1c13948 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7de9641318e646c2a322d87765a4fad4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:32:22.362905 6641 consensus_queue.cc:1035] T 66bf7ad027c14aae8349191197dc6958 P 587e8b5a872f472c91f40242e1c13948 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e30f237517411ea5e54c172fb7c268" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 37927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:32:22.369817 6647 mvcc.cc:205] Tried to move back new op lower bound from 7298578605506363392 to 7298578605369704448. Current Snapshot: MvccSnapshot[applied={T|T < 7298578605506363392}]
I20260619 14:32:22.372778 6648 mvcc.cc:205] Tried to move back new op lower bound from 7298578605506363392 to 7298578605369704448. Current Snapshot: MvccSnapshot[applied={T|T < 7298578605506363392}]
I20260619 14:32:22.379854 6157 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:32:22.380057 6157 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:32:22.382623 6157 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 13d21fe2120149a39823873153a21d36: Sending DeleteTablet for 3 replicas of tablet 66bf7ad027c14aae8349191197dc6958
I20260619 14:32:22.383567 6551 tablet_service.cc:1514] Processing DeleteTablet for tablet 66bf7ad027c14aae8349191197dc6958 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:32:22 UTC) from {username='kudu', principal='kudu/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:38504
I20260619 14:32:22.383569 6418 tablet_service.cc:1514] Processing DeleteTablet for tablet 66bf7ad027c14aae8349191197dc6958 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:32:22 UTC) from {username='kudu', principal='kudu/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:42322
I20260619 14:32:22.383651 6285 tablet_service.cc:1514] Processing DeleteTablet for tablet 66bf7ad027c14aae8349191197dc6958 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:32:22 UTC) from {username='kudu', principal='kudu/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:42754
I20260619 14:32:22.384119 6657 tablet_replica.cc:330] T 66bf7ad027c14aae8349191197dc6958 P 30e30f237517411ea5e54c172fb7c268: stopping tablet replica
I20260619 14:32:22.384181 6659 tablet_replica.cc:330] T 66bf7ad027c14aae8349191197dc6958 P 7de9641318e646c2a322d87765a4fad4: stopping tablet replica
I20260619 14:32:22.384356 6658 tablet_replica.cc:330] T 66bf7ad027c14aae8349191197dc6958 P 587e8b5a872f472c91f40242e1c13948: stopping tablet replica
I20260619 14:32:22.384467 6659 raft_consensus.cc:2229] T 66bf7ad027c14aae8349191197dc6958 P 7de9641318e646c2a322d87765a4fad4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:32:22.384467 6657 raft_consensus.cc:2229] T 66bf7ad027c14aae8349191197dc6958 P 30e30f237517411ea5e54c172fb7c268 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:32:22.384639 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 6223
I20260619 14:32:22.384807 6657 raft_consensus.cc:2258] T 66bf7ad027c14aae8349191197dc6958 P 30e30f237517411ea5e54c172fb7c268 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:32:22.384809 6659 raft_consensus.cc:2258] T 66bf7ad027c14aae8349191197dc6958 P 7de9641318e646c2a322d87765a4fad4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:32:22.384922 6658 raft_consensus.cc:2229] T 66bf7ad027c14aae8349191197dc6958 P 587e8b5a872f472c91f40242e1c13948 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:32:22.385318 6658 raft_consensus.cc:2258] T 66bf7ad027c14aae8349191197dc6958 P 587e8b5a872f472c91f40242e1c13948 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:32:22.387066 6657 ts_tablet_manager.cc:1905] T 66bf7ad027c14aae8349191197dc6958 P 30e30f237517411ea5e54c172fb7c268: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:32:22.388007 6658 ts_tablet_manager.cc:1905] T 66bf7ad027c14aae8349191197dc6958 P 587e8b5a872f472c91f40242e1c13948: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:32:22.389847 6657 ts_tablet_manager.cc:1918] T 66bf7ad027c14aae8349191197dc6958 P 30e30f237517411ea5e54c172fb7c268: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:32:22.389958 6657 log.cc:1196] T 66bf7ad027c14aae8349191197dc6958 P 30e30f237517411ea5e54c172fb7c268: Deleting WAL directory at /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/ts-1/wal/wals/66bf7ad027c14aae8349191197dc6958
I20260619 14:32:22.390326 6657 ts_tablet_manager.cc:1939] T 66bf7ad027c14aae8349191197dc6958 P 30e30f237517411ea5e54c172fb7c268: Deleting consensus metadata
I20260619 14:32:22.391105 6658 ts_tablet_manager.cc:1918] T 66bf7ad027c14aae8349191197dc6958 P 587e8b5a872f472c91f40242e1c13948: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:32:22.391274 6658 log.cc:1196] T 66bf7ad027c14aae8349191197dc6958 P 587e8b5a872f472c91f40242e1c13948: Deleting WAL directory at /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879514993456-2056-0/minicluster-data/ts-2/wal/wals/66bf7ad027c14aae8349191197dc6958
I20260619 14:32:22.391654 6658 ts_tablet_manager.cc:1939] T 66bf7ad027c14aae8349191197dc6958 P 587e8b5a872f472c91f40242e1c13948: Deleting consensus metadata
I20260619 14:32:22.391806 6148 catalog_manager.cc:4841] TS 30e30f237517411ea5e54c172fb7c268 (127.2.2.2:37927): tablet 66bf7ad027c14aae8349191197dc6958 (table test-table [id=02ad1a87b2fa46e2a9008505bf3170b7]) successfully deleted
W20260619 14:32:22.392896 6146 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.2.1:37947 (error 108)
I20260619 14:32:22.393115 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 6356
I20260619 14:32:22.393386 6146 catalog_manager.cc:4841] TS 587e8b5a872f472c91f40242e1c13948 (127.2.2.3:45747): tablet 66bf7ad027c14aae8349191197dc6958 (table test-table [id=02ad1a87b2fa46e2a9008505bf3170b7]) successfully deleted
I20260619 14:32:22.395551 6632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:22.395273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.1:37947 (local address 127.0.0.1:42764)
0619 14:32:22.395407 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:22.395478 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.1:37947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 14:32:22.395766 6146 catalog_manager.cc:4584] TS 7de9641318e646c2a322d87765a4fad4 (127.2.2.1:37947): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 66bf7ad027c14aae8349191197dc6958: Network error: Client connection negotiation failed: client connection to 127.2.2.1:37947: connect: Connection refused (error 111)
I20260619 14:32:22.400456 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 6489
I20260619 14:32:22.405257 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 6130
2026-06-19T14:32:22Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2318 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-19T14:32:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:32:22Z Disabled control of system clock
I20260619 14:32:22.436115 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:36013
--webserver_interface=127.2.2.62
--webserver_port=0
--builtin_ntp_servers=127.2.2.20:37949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.2.62:36013
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 14:32:22.548038 6664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:22.548363 6664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:22.548403 6664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:22.552753 6664 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:32:22.552829 6664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:22.552845 6664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:22.552863 6664 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:32:22.552879 6664 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:32:22.558058 6664 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:37949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.2.62:36013
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:36013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.2.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:32:22.559235 6664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:22.560672 6664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:32:22.569329 6670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:22.569372 6669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:22.569483 6664 server_base.cc:1034] running on GCE node
W20260619 14:32:22.569353 6672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:22.570248 6664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:22.571362 6664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:22.572557 6664 hybrid_clock.cc:648] HybridClock initialized: now 1781879542572543 us; error 28 us; skew 500 ppm
I20260619 14:32:22.578976 6664 webserver.cc:460] Webserver started at http://127.2.2.62:33963/ using document root <none> and password file <none>
I20260619 14:32:22.579666 6664 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:22.579746 6664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:22.579986 6664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:32:22.582345 6664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/master-0/data/instance:
uuid: "5caedc2665cc4ad3867d80443a2c9d7f"
format_stamp: "Formatted at 2026-06-19 14:32:22 on dist-test-slave-70rm"
server_key: "b594066b62aef06e1f966bb922eb581a"
server_key_iv: "f35d7ffb12b03f7c24420ec5c14f366b"
server_key_version: "encryptionkey@0"
I20260619 14:32:22.582940 6664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/master-0/wal/instance:
uuid: "5caedc2665cc4ad3867d80443a2c9d7f"
format_stamp: "Formatted at 2026-06-19 14:32:22 on dist-test-slave-70rm"
server_key: "b594066b62aef06e1f966bb922eb581a"
server_key_iv: "f35d7ffb12b03f7c24420ec5c14f366b"
server_key_version: "encryptionkey@0"
I20260619 14:32:22.586776 6664 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 14:32:22.589301 6678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:22.590745 6664 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:32:22.590898 6664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/master-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/master-0/wal
uuid: "5caedc2665cc4ad3867d80443a2c9d7f"
format_stamp: "Formatted at 2026-06-19 14:32:22 on dist-test-slave-70rm"
server_key: "b594066b62aef06e1f966bb922eb581a"
server_key_iv: "f35d7ffb12b03f7c24420ec5c14f366b"
server_key_version: "encryptionkey@0"
I20260619 14:32:22.591025 6664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:22.595908 6664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:22.596596 6664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:32:22.596797 6664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:22.603855 6664 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.62:36013
I20260619 14:32:22.603859 6730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.62:36013 every 8 connection(s)
I20260619 14:32:22.604813 6664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
I20260619 14:32:22.608078 6731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:22.612262 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 6664
I20260619 14:32:22.612370 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/master-0/wal/instance
I20260619 14:32:22.612649 2056 external_mini_cluster.cc:1428] Setting key 9fbe2c414884da4435bc419308c17230
I20260619 14:32:22.614368 6731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5caedc2665cc4ad3867d80443a2c9d7f: Bootstrap starting.
I20260619 14:32:22.616837 6731 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5caedc2665cc4ad3867d80443a2c9d7f: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:22.617811 6731 log.cc:824] T 00000000000000000000000000000000 P 5caedc2665cc4ad3867d80443a2c9d7f: Log is configured to *not* fsync() on all Append() calls
I20260619 14:32:22.618606 6734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:22.614238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33576 (local address 127.2.2.62:36013)
0619 14:32:22.614878 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:22.614894 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:22.614933 (+ 39us) server_negotiation.cc:407] Connection header received
0619 14:32:22.615591 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:22.615609 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:22.616008 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:22.616308 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:22.617173 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:22.617178 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:22.617191 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:32:22.617202 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:22.617260 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:32:22.617278 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:22.617366 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:22.617505 (+ 139us) server_negotiation.cc:299] Negotiation successful
0619 14:32:22.617790 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":126,"threads_started":1}
I20260619 14:32:22.619941 6731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5caedc2665cc4ad3867d80443a2c9d7f: No bootstrap required, opened a new log
I20260619 14:32:22.623903 6731 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5caedc2665cc4ad3867d80443a2c9d7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5caedc2665cc4ad3867d80443a2c9d7f" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 36013 } }
I20260619 14:32:22.624156 6731 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5caedc2665cc4ad3867d80443a2c9d7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:22.624229 6731 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5caedc2665cc4ad3867d80443a2c9d7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5caedc2665cc4ad3867d80443a2c9d7f, State: Initialized, Role: FOLLOWER
I20260619 14:32:22.624646 6731 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5caedc2665cc4ad3867d80443a2c9d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5caedc2665cc4ad3867d80443a2c9d7f" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 36013 } }
I20260619 14:32:22.624788 6731 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5caedc2665cc4ad3867d80443a2c9d7f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:32:22.624852 6731 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5caedc2665cc4ad3867d80443a2c9d7f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:32:22.624981 6731 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5caedc2665cc4ad3867d80443a2c9d7f [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:22.625917 6731 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5caedc2665cc4ad3867d80443a2c9d7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5caedc2665cc4ad3867d80443a2c9d7f" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 36013 } }
I20260619 14:32:22.626309 6731 leader_election.cc:302] T 00000000000000000000000000000000 P 5caedc2665cc4ad3867d80443a2c9d7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5caedc2665cc4ad3867d80443a2c9d7f; no voters:
I20260619 14:32:22.626669 6731 leader_election.cc:288] T 00000000000000000000000000000000 P 5caedc2665cc4ad3867d80443a2c9d7f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:32:22.626767 6736 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5caedc2665cc4ad3867d80443a2c9d7f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:32:22.626945 6736 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5caedc2665cc4ad3867d80443a2c9d7f [term 1 LEADER]: Becoming Leader. State: Replica: 5caedc2665cc4ad3867d80443a2c9d7f, State: Running, Role: LEADER
I20260619 14:32:22.627243 6736 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5caedc2665cc4ad3867d80443a2c9d7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5caedc2665cc4ad3867d80443a2c9d7f" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 36013 } }
I20260619 14:32:22.628037 6731 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5caedc2665cc4ad3867d80443a2c9d7f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:32:22.628943 6738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5caedc2665cc4ad3867d80443a2c9d7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5caedc2665cc4ad3867d80443a2c9d7f. Latest consensus state: current_term: 1 leader_uuid: "5caedc2665cc4ad3867d80443a2c9d7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5caedc2665cc4ad3867d80443a2c9d7f" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 36013 } } }
I20260619 14:32:22.629081 6738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5caedc2665cc4ad3867d80443a2c9d7f [sys.catalog]: This master's current role is: LEADER
I20260619 14:32:22.629380 6737 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5caedc2665cc4ad3867d80443a2c9d7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5caedc2665cc4ad3867d80443a2c9d7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5caedc2665cc4ad3867d80443a2c9d7f" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 36013 } } }
I20260619 14:32:22.629469 6737 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5caedc2665cc4ad3867d80443a2c9d7f [sys.catalog]: This master's current role is: LEADER
I20260619 14:32:22.629599 6745 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:32:22.632764 6745 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:32:22.638418 6745 catalog_manager.cc:1347] Generated new cluster ID: 415b7f90bfab4ab2a4fa33d4f5d8b006
I20260619 14:32:22.638502 6745 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:32:22.667330 6745 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:32:22.667708 6745 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:32:22.677358 6745 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5caedc2665cc4ad3867d80443a2c9d7f: Generated new TSK 0
I20260619 14:32:22.678207 6745 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:32:22.687743 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.1:0
--local_ip_for_outbound_sockets=127.2.2.1
--webserver_interface=127.2.2.1
--webserver_port=0
--tserver_master_addrs=127.2.2.62:36013
--builtin_ntp_servers=127.2.2.20:37949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 14:32:22.803692 6755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:22.803982 6755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:22.804076 6755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:22.808216 6755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:22.808282 6755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:22.808398 6755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.1
I20260619 14:32:22.813235 6755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:37949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.2.1
--webserver_port=0
--tserver_master_addrs=127.2.2.62:36013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.1
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:32:22.814385 6755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:22.815820 6755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:32:22.825456 6761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:22.825553 6755 server_base.cc:1034] running on GCE node
W20260619 14:32:22.825471 6760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:22.825471 6763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:22.826236 6755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:22.826918 6755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:22.828176 6755 hybrid_clock.cc:648] HybridClock initialized: now 1781879542828150 us; error 38 us; skew 500 ppm
I20260619 14:32:22.830250 6755 webserver.cc:460] Webserver started at http://127.2.2.1:44387/ using document root <none> and password file <none>
I20260619 14:32:22.830850 6755 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:22.830942 6755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:22.831161 6755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:32:22.833232 6755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/ts-0/data/instance:
uuid: "948a96a173de40c9895d9ce3b9af99e5"
format_stamp: "Formatted at 2026-06-19 14:32:22 on dist-test-slave-70rm"
server_key: "86b0740a3bf89086f96b169a8212d8ec"
server_key_iv: "643d152824b2398c155138c58f7a010c"
server_key_version: "encryptionkey@0"
I20260619 14:32:22.833819 6755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/ts-0/wal/instance:
uuid: "948a96a173de40c9895d9ce3b9af99e5"
format_stamp: "Formatted at 2026-06-19 14:32:22 on dist-test-slave-70rm"
server_key: "86b0740a3bf89086f96b169a8212d8ec"
server_key_iv: "643d152824b2398c155138c58f7a010c"
server_key_version: "encryptionkey@0"
I20260619 14:32:22.837575 6755 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 14:32:22.840124 6769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:22.841374 6755 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:32:22.841509 6755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/ts-0/wal
uuid: "948a96a173de40c9895d9ce3b9af99e5"
format_stamp: "Formatted at 2026-06-19 14:32:22 on dist-test-slave-70rm"
server_key: "86b0740a3bf89086f96b169a8212d8ec"
server_key_iv: "643d152824b2398c155138c58f7a010c"
server_key_version: "encryptionkey@0"
I20260619 14:32:22.841629 6755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:22.846376 6755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:22.846977 6755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:32:22.847153 6755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:22.847687 6755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:32:22.848623 6755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:32:22.848688 6755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:22.848755 6755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:32:22.848804 6755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:22.857398 6755 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.1:37905
I20260619 14:32:22.857445 6882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.1:37905 every 8 connection(s)
I20260619 14:32:22.858418 6755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
I20260619 14:32:22.864248 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 6755
I20260619 14:32:22.864393 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/ts-0/wal/instance
I20260619 14:32:22.864694 2056 external_mini_cluster.cc:1428] Setting key ac9a5e2011d2baacd3413cb0a838f2c6
I20260619 14:32:22.865285 6734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:22.860577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:56031 (local address 127.2.2.62:36013)
0619 14:32:22.860750 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:22.860754 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:22.861908 (+ 1154us) server_negotiation.cc:407] Connection header received
0619 14:32:22.863444 (+ 1536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:22.863448 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:22.863493 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:22.863576 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:22.864786 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:22.864789 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:22.864792 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:32:22.864805 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:22.864828 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:32:22.864839 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:22.864942 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:22.865102 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 14:32:22.865141 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 14:32:22.866345 6885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:22.861023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:36013 (local address 127.2.2.1:56031)
0619 14:32:22.861746 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:22.861788 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:22.863227 (+ 1439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:22.863749 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:22.863757 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:22.863846 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:22.864503 (+ 657us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:32:22.864506 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:22.864565 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:32:22.864642 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:22.864932 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:22.864940 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:22.864946 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 14:32:22.865146 (+ 200us) client_negotiation.cc:240] Negotiation successful
0619 14:32:22.865423 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":128,"threads_started":1}
I20260619 14:32:22.866683 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.2:0
--local_ip_for_outbound_sockets=127.2.2.2
--webserver_interface=127.2.2.2
--webserver_port=0
--tserver_master_addrs=127.2.2.62:36013
--builtin_ntp_servers=127.2.2.20:37949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 14:32:22.867774 6883 heartbeater.cc:347] Connected to a master server at 127.2.2.62:36013
I20260619 14:32:22.868073 6883 heartbeater.cc:464] Registering TS with master...
I20260619 14:32:22.868517 6883 heartbeater.cc:511] Master 127.2.2.62:36013 requested a full tablet report, sending...
I20260619 14:32:22.870005 6695 ts_manager.cc:194] Registered new tserver with Master: 948a96a173de40c9895d9ce3b9af99e5 (127.2.2.1:37905)
W20260619 14:32:22.978417 6886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:22.978695 6886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:22.978763 6886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:22.982921 6886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:22.982976 6886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:22.983033 6886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.2
I20260619 14:32:22.987838 6886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:37949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.2.2
--webserver_port=0
--tserver_master_addrs=127.2.2.62:36013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.2
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:32:22.988930 6886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:22.990360 6886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:32:23.000344 6892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:23.000295 6894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:23.000300 6891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:23.000734 6886 server_base.cc:1034] running on GCE node
I20260619 14:32:23.001148 6886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:23.001816 6886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:23.002975 6886 hybrid_clock.cc:648] HybridClock initialized: now 1781879543002934 us; error 58 us; skew 500 ppm
I20260619 14:32:23.004802 6886 webserver.cc:460] Webserver started at http://127.2.2.2:37891/ using document root <none> and password file <none>
I20260619 14:32:23.005402 6886 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:23.005467 6886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:23.005697 6886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:32:23.007788 6886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/ts-1/data/instance:
uuid: "b105fff72c0b44bc829fde8dfe4c2752"
format_stamp: "Formatted at 2026-06-19 14:32:22 on dist-test-slave-70rm"
server_key: "2aea9e92ae4ed3e7ce1a59447ea5fb58"
server_key_iv: "e7375f0fa019af4cab8b9ae522e6fc5a"
server_key_version: "encryptionkey@0"
I20260619 14:32:23.008354 6886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/ts-1/wal/instance:
uuid: "b105fff72c0b44bc829fde8dfe4c2752"
format_stamp: "Formatted at 2026-06-19 14:32:22 on dist-test-slave-70rm"
server_key: "2aea9e92ae4ed3e7ce1a59447ea5fb58"
server_key_iv: "e7375f0fa019af4cab8b9ae522e6fc5a"
server_key_version: "encryptionkey@0"
I20260619 14:32:23.012086 6886 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260619 14:32:23.014669 6900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:23.015930 6886 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:32:23.016070 6886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/ts-1/wal
uuid: "b105fff72c0b44bc829fde8dfe4c2752"
format_stamp: "Formatted at 2026-06-19 14:32:22 on dist-test-slave-70rm"
server_key: "2aea9e92ae4ed3e7ce1a59447ea5fb58"
server_key_iv: "e7375f0fa019af4cab8b9ae522e6fc5a"
server_key_version: "encryptionkey@0"
I20260619 14:32:23.016184 6886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:23.020779 6886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:23.021391 6886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:32:23.021586 6886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:23.022307 6886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:32:23.023216 6886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:32:23.023262 6886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:23.023300 6886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:32:23.023315 6886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:23.032286 6886 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.2:34179
I20260619 14:32:23.032310 7013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.2:34179 every 8 connection(s)
I20260619 14:32:23.033229 6886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
I20260619 14:32:23.039463 6734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:23.035040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:55811 (local address 127.2.2.62:36013)
0619 14:32:23.035186 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:23.035191 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:23.036239 (+ 1048us) server_negotiation.cc:407] Connection header received
0619 14:32:23.037603 (+ 1364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:23.037607 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:23.037652 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:23.037765 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:23.038969 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:23.038973 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:23.038975 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:32:23.038988 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:23.039018 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:32:23.039027 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:23.039123 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:23.039277 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 14:32:23.039323 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 14:32:23.040302 7016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:23.035435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:36013 (local address 127.2.2.2:55811)
0619 14:32:23.036085 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:23.036127 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:23.037371 (+ 1244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:23.037936 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:23.037943 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:23.038039 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:23.038698 (+ 659us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:32:23.038700 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:23.038745 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:32:23.038814 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:23.039115 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:23.039123 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:23.039131 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 14:32:23.039327 (+ 196us) client_negotiation.cc:240] Negotiation successful
0619 14:32:23.039570 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":112,"threads_started":1}
I20260619 14:32:23.041399 7014 heartbeater.cc:347] Connected to a master server at 127.2.2.62:36013
I20260619 14:32:23.041638 7014 heartbeater.cc:464] Registering TS with master...
I20260619 14:32:23.042044 7014 heartbeater.cc:511] Master 127.2.2.62:36013 requested a full tablet report, sending...
I20260619 14:32:23.043030 6695 ts_manager.cc:194] Registered new tserver with Master: b105fff72c0b44bc829fde8dfe4c2752 (127.2.2.2:34179)
I20260619 14:32:23.043056 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 6886
I20260619 14:32:23.043182 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/ts-1/wal/instance
I20260619 14:32:23.043485 2056 external_mini_cluster.cc:1428] Setting key 00c0b4b88464f9cde430736e548fd172
I20260619 14:32:23.045235 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.3:0
--local_ip_for_outbound_sockets=127.2.2.3
--webserver_interface=127.2.2.3
--webserver_port=0
--tserver_master_addrs=127.2.2.62:36013
--builtin_ntp_servers=127.2.2.20:37949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 14:32:23.157935 7017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:23.158274 7017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:23.158317 7017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:23.162568 7017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:23.162638 7017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:23.162698 7017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.3
I20260619 14:32:23.167589 7017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:37949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.2.3
--webserver_port=0
--tserver_master_addrs=127.2.2.62:36013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.3
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:32:23.168711 7017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:23.170094 7017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:32:23.180482 7023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:23.180466 7022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:23.180466 7025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:23.180889 7017 server_base.cc:1034] running on GCE node
I20260619 14:32:23.181286 7017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:23.181937 7017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:23.183120 7017 hybrid_clock.cc:648] HybridClock initialized: now 1781879543183100 us; error 38 us; skew 500 ppm
I20260619 14:32:23.185225 7017 webserver.cc:460] Webserver started at http://127.2.2.3:38543/ using document root <none> and password file <none>
I20260619 14:32:23.185858 7017 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:23.185927 7017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:23.186196 7017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:32:23.188413 7017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/ts-2/data/instance:
uuid: "f7469aaaedb845309056ff86fdb36244"
format_stamp: "Formatted at 2026-06-19 14:32:23 on dist-test-slave-70rm"
server_key: "36ef18fe39485f62d7901f57f6bec964"
server_key_iv: "8329e48f76281d93ecca1459674d76dc"
server_key_version: "encryptionkey@0"
I20260619 14:32:23.189046 7017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/ts-2/wal/instance:
uuid: "f7469aaaedb845309056ff86fdb36244"
format_stamp: "Formatted at 2026-06-19 14:32:23 on dist-test-slave-70rm"
server_key: "36ef18fe39485f62d7901f57f6bec964"
server_key_iv: "8329e48f76281d93ecca1459674d76dc"
server_key_version: "encryptionkey@0"
I20260619 14:32:23.193226 7017 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260619 14:32:23.195950 7031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:23.197415 7017 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 14:32:23.197543 7017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/ts-2/wal
uuid: "f7469aaaedb845309056ff86fdb36244"
format_stamp: "Formatted at 2026-06-19 14:32:23 on dist-test-slave-70rm"
server_key: "36ef18fe39485f62d7901f57f6bec964"
server_key_iv: "8329e48f76281d93ecca1459674d76dc"
server_key_version: "encryptionkey@0"
I20260619 14:32:23.197713 7017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:23.202679 7017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:23.203346 7017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:32:23.203543 7017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:23.204128 7017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:32:23.205093 7017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:32:23.205139 7017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:23.205180 7017 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:32:23.205228 7017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:23.213600 7017 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.3:35781
I20260619 14:32:23.213621 7144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.3:35781 every 8 connection(s)
I20260619 14:32:23.214607 7017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
I20260619 14:32:23.220806 6734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:23.216416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.3:58763 (local address 127.2.2.62:36013)
0619 14:32:23.216590 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:23.216594 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:23.217666 (+ 1072us) server_negotiation.cc:407] Connection header received
0619 14:32:23.219058 (+ 1392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:23.219061 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:23.219107 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:23.219218 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:23.220311 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:23.220314 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:23.220317 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:32:23.220329 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:23.220358 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:32:23.220366 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:23.220468 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:23.220610 (+ 142us) server_negotiation.cc:299] Negotiation successful
0619 14:32:23.220657 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 14:32:23.221427 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 7017
I20260619 14:32:23.221531 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/ts-2/wal/instance
I20260619 14:32:23.221827 2056 external_mini_cluster.cc:1428] Setting key 1cc532d413627548fdba357ddc94e34e
I20260619 14:32:23.221920 7147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:23.216835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:36013 (local address 127.2.2.3:58763)
0619 14:32:23.217510 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:23.217551 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:23.218826 (+ 1275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:23.219395 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:23.219402 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:23.219493 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:23.220043 (+ 550us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:32:23.220044 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:23.220084 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:32:23.220153 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:23.220450 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:23.220458 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:23.220463 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 14:32:23.220647 (+ 184us) client_negotiation.cc:240] Negotiation successful
0619 14:32:23.220895 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":115,"threads_started":1}
I20260619 14:32:23.223269 7145 heartbeater.cc:347] Connected to a master server at 127.2.2.62:36013
I20260619 14:32:23.223516 7145 heartbeater.cc:464] Registering TS with master...
I20260619 14:32:23.223982 7145 heartbeater.cc:511] Master 127.2.2.62:36013 requested a full tablet report, sending...
I20260619 14:32:23.225049 6695 ts_manager.cc:194] Registered new tserver with Master: f7469aaaedb845309056ff86fdb36244 (127.2.2.3:35781)
I20260619 14:32:23.235868 2056 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:32:23.239530 6734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:23.237821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33578 (local address 127.2.2.62:36013)
0619 14:32:23.237988 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:23.237992 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:23.238142 (+ 150us) server_negotiation.cc:407] Connection header received
0619 14:32:23.238299 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:23.238301 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:23.238337 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:23.238396 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:23.239083 (+ 687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:23.239086 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:23.239088 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:32:23.239100 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:23.239122 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:32:23.239131 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:23.239215 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:23.239274 (+ 59us) server_negotiation.cc:299] Negotiation successful
0619 14:32:23.239310 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 14:32:23.242386 6695 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33578:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:32:23.245553 6695 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:32:23.257628 7157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:23.254387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.2:34179 (local address 127.0.0.1:59522)
0619 14:32:23.255294 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:23.255351 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:23.255454 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:23.256094 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:23.256102 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:23.256194 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:23.256842 (+ 648us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:32:23.256844 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:23.256867 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:32:23.256938 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:23.257243 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:23.257254 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:23.257261 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 14:32:23.257363 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 14:32:23.257433 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":715,"thread_start_us":124,"threads_started":1}
I20260619 14:32:23.258282 7159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:23.255147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59522 (local address 127.2.2.2:34179)
0619 14:32:23.255598 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:23.255608 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:23.255628 (+ 20us) server_negotiation.cc:407] Connection header received
0619 14:32:23.255700 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:23.255707 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:23.255918 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:23.256070 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:23.257038 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:23.257043 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:23.257052 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:32:23.257060 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:23.257091 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:32:23.257107 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:23.257935 (+ 828us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:23.257990 (+ 55us) server_negotiation.cc:299] Negotiation successful
0619 14:32:23.258069 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":135,"threads_started":1}
I20260619 14:32:23.258548 7156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:23.254628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.3:35781 (local address 127.0.0.1:37364)
0619 14:32:23.255970 (+ 1342us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:23.255986 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:23.256058 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:23.257185 (+ 1127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:23.257190 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:23.257208 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:23.257646 (+ 438us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:32:23.257650 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:23.257662 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:32:23.257937 (+ 275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:23.258319 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:23.258325 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:23.258331 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 14:32:23.258397 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 14:32:23.258433 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1275,"thread_start_us":89,"threads_started":1}
I20260619 14:32:23.258867 7161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:23.255501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40018 (local address 127.2.2.1:37905)
0619 14:32:23.255839 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:23.255850 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:23.256335 (+ 485us) server_negotiation.cc:407] Connection header received
0619 14:32:23.256406 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:23.256414 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:23.256596 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:23.256812 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:23.258161 (+ 1349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:23.258168 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:23.258179 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:32:23.258193 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:23.258240 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:32:23.258280 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:23.258376 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:23.258598 (+ 222us) server_negotiation.cc:299] Negotiation successful
0619 14:32:23.258673 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":128,"threads_started":1}
I20260619 14:32:23.259078 7158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:23.254797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37364 (local address 127.2.2.3:35781)
0619 14:32:23.255922 (+ 1125us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:23.255938 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:23.256499 (+ 561us) server_negotiation.cc:407] Connection header received
0619 14:32:23.256559 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:23.256565 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:23.256759 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:23.257192 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:23.258070 (+ 878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:23.258075 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:23.258086 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:32:23.258098 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:23.258175 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:32:23.258211 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:23.258824 (+ 613us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:23.258867 (+ 43us) server_negotiation.cc:299] Negotiation successful
0619 14:32:23.258926 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":998,"thread_start_us":158,"threads_started":1}
I20260619 14:32:23.259240 7160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:23.255339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.1:37905 (local address 127.0.0.1:40018)
0619 14:32:23.256216 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:23.256228 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:23.256306 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:23.257490 (+ 1184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:23.257494 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:23.257514 (+ 20us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:23.257929 (+ 415us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:32:23.257931 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:23.257944 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:32:23.257973 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:23.258456 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:23.258459 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:23.258462 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 14:32:23.259074 (+ 612us) client_negotiation.cc:240] Negotiation successful
0619 14:32:23.259116 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"thread_start_us":104,"threads_started":1}
I20260619 14:32:23.261204 6948 tablet_service.cc:1467] Processing CreateTablet for tablet 50f19b2dbbcc44b2b09af4a5dad8fc14 (DEFAULT_TABLE table=test-table [id=d434fa88f55b46b3b6128caf8fd803f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:32:23.262354 6948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50f19b2dbbcc44b2b09af4a5dad8fc14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:23.262014 7079 tablet_service.cc:1467] Processing CreateTablet for tablet 50f19b2dbbcc44b2b09af4a5dad8fc14 (DEFAULT_TABLE table=test-table [id=d434fa88f55b46b3b6128caf8fd803f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:32:23.262017 6817 tablet_service.cc:1467] Processing CreateTablet for tablet 50f19b2dbbcc44b2b09af4a5dad8fc14 (DEFAULT_TABLE table=test-table [id=d434fa88f55b46b3b6128caf8fd803f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:32:23.263062 7079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50f19b2dbbcc44b2b09af4a5dad8fc14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:23.263062 6817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50f19b2dbbcc44b2b09af4a5dad8fc14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:23.269366 7162 tablet_bootstrap.cc:492] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P b105fff72c0b44bc829fde8dfe4c2752: Bootstrap starting.
I20260619 14:32:23.271486 7164 tablet_bootstrap.cc:492] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P 948a96a173de40c9895d9ce3b9af99e5: Bootstrap starting.
I20260619 14:32:23.272281 7162 tablet_bootstrap.cc:654] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P b105fff72c0b44bc829fde8dfe4c2752: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:23.273049 7162 log.cc:824] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P b105fff72c0b44bc829fde8dfe4c2752: Log is configured to *not* fsync() on all Append() calls
I20260619 14:32:23.273384 7164 tablet_bootstrap.cc:654] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P 948a96a173de40c9895d9ce3b9af99e5: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:23.273784 7163 tablet_bootstrap.cc:492] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P f7469aaaedb845309056ff86fdb36244: Bootstrap starting.
I20260619 14:32:23.274194 7164 log.cc:824] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P 948a96a173de40c9895d9ce3b9af99e5: Log is configured to *not* fsync() on all Append() calls
I20260619 14:32:23.275272 7162 tablet_bootstrap.cc:492] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P b105fff72c0b44bc829fde8dfe4c2752: No bootstrap required, opened a new log
I20260619 14:32:23.275466 7162 ts_tablet_manager.cc:1397] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P b105fff72c0b44bc829fde8dfe4c2752: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 14:32:23.276083 7163 tablet_bootstrap.cc:654] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P f7469aaaedb845309056ff86fdb36244: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:23.276222 7164 tablet_bootstrap.cc:492] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P 948a96a173de40c9895d9ce3b9af99e5: No bootstrap required, opened a new log
I20260619 14:32:23.276400 7164 ts_tablet_manager.cc:1397] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P 948a96a173de40c9895d9ce3b9af99e5: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 14:32:23.276826 7163 log.cc:824] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P f7469aaaedb845309056ff86fdb36244: Log is configured to *not* fsync() on all Append() calls
I20260619 14:32:23.278501 7163 tablet_bootstrap.cc:492] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P f7469aaaedb845309056ff86fdb36244: No bootstrap required, opened a new log
I20260619 14:32:23.278697 7163 ts_tablet_manager.cc:1397] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P f7469aaaedb845309056ff86fdb36244: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:32:23.280295 7162 raft_consensus.cc:348] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P b105fff72c0b44bc829fde8dfe4c2752 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b105fff72c0b44bc829fde8dfe4c2752" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 34179 } } peers { permanent_uuid: "f7469aaaedb845309056ff86fdb36244" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 35781 } } peers { permanent_uuid: "948a96a173de40c9895d9ce3b9af99e5" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37905 } }
I20260619 14:32:23.280591 7162 raft_consensus.cc:374] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P b105fff72c0b44bc829fde8dfe4c2752 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:23.280663 7162 raft_consensus.cc:729] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P b105fff72c0b44bc829fde8dfe4c2752 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b105fff72c0b44bc829fde8dfe4c2752, State: Initialized, Role: FOLLOWER
I20260619 14:32:23.280915 7164 raft_consensus.cc:348] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P 948a96a173de40c9895d9ce3b9af99e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b105fff72c0b44bc829fde8dfe4c2752" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 34179 } } peers { permanent_uuid: "f7469aaaedb845309056ff86fdb36244" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 35781 } } peers { permanent_uuid: "948a96a173de40c9895d9ce3b9af99e5" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37905 } }
I20260619 14:32:23.281172 7164 raft_consensus.cc:374] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P 948a96a173de40c9895d9ce3b9af99e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:23.281214 7164 raft_consensus.cc:729] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P 948a96a173de40c9895d9ce3b9af99e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948a96a173de40c9895d9ce3b9af99e5, State: Initialized, Role: FOLLOWER
I20260619 14:32:23.281140 7162 consensus_queue.cc:260] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P b105fff72c0b44bc829fde8dfe4c2752 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b105fff72c0b44bc829fde8dfe4c2752" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 34179 } } peers { permanent_uuid: "f7469aaaedb845309056ff86fdb36244" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 35781 } } peers { permanent_uuid: "948a96a173de40c9895d9ce3b9af99e5" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37905 } }
I20260619 14:32:23.281682 7164 consensus_queue.cc:260] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P 948a96a173de40c9895d9ce3b9af99e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b105fff72c0b44bc829fde8dfe4c2752" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 34179 } } peers { permanent_uuid: "f7469aaaedb845309056ff86fdb36244" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 35781 } } peers { permanent_uuid: "948a96a173de40c9895d9ce3b9af99e5" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37905 } }
I20260619 14:32:23.282219 7162 ts_tablet_manager.cc:1428] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P b105fff72c0b44bc829fde8dfe4c2752: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260619 14:32:23.282584 7014 heartbeater.cc:503] Master 127.2.2.62:36013 was elected leader, sending a full tablet report...
I20260619 14:32:23.282648 7164 ts_tablet_manager.cc:1428] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P 948a96a173de40c9895d9ce3b9af99e5: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 14:32:23.282742 6883 heartbeater.cc:503] Master 127.2.2.62:36013 was elected leader, sending a full tablet report...
I20260619 14:32:23.283507 7163 raft_consensus.cc:348] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P f7469aaaedb845309056ff86fdb36244 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b105fff72c0b44bc829fde8dfe4c2752" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 34179 } } peers { permanent_uuid: "f7469aaaedb845309056ff86fdb36244" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 35781 } } peers { permanent_uuid: "948a96a173de40c9895d9ce3b9af99e5" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37905 } }
I20260619 14:32:23.283823 7163 raft_consensus.cc:374] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P f7469aaaedb845309056ff86fdb36244 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:23.283883 7163 raft_consensus.cc:729] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P f7469aaaedb845309056ff86fdb36244 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7469aaaedb845309056ff86fdb36244, State: Initialized, Role: FOLLOWER
W20260619 14:32:23.284113 7015 tablet.cc:2369] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P b105fff72c0b44bc829fde8dfe4c2752: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:32:23.284330 7163 consensus_queue.cc:260] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P f7469aaaedb845309056ff86fdb36244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b105fff72c0b44bc829fde8dfe4c2752" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 34179 } } peers { permanent_uuid: "f7469aaaedb845309056ff86fdb36244" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 35781 } } peers { permanent_uuid: "948a96a173de40c9895d9ce3b9af99e5" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37905 } }
I20260619 14:32:23.284993 7145 heartbeater.cc:503] Master 127.2.2.62:36013 was elected leader, sending a full tablet report...
I20260619 14:32:23.285323 7163 ts_tablet_manager.cc:1428] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P f7469aaaedb845309056ff86fdb36244: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
W20260619 14:32:23.359723 6884 tablet.cc:2369] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P 948a96a173de40c9895d9ce3b9af99e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:32:23.376824 7168 raft_consensus.cc:482] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P b105fff72c0b44bc829fde8dfe4c2752 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:32:23.377049 7168 raft_consensus.cc:504] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P b105fff72c0b44bc829fde8dfe4c2752 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b105fff72c0b44bc829fde8dfe4c2752" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 34179 } } peers { permanent_uuid: "f7469aaaedb845309056ff86fdb36244" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 35781 } } peers { permanent_uuid: "948a96a173de40c9895d9ce3b9af99e5" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37905 } }
I20260619 14:32:23.378264 7168 leader_election.cc:288] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P b105fff72c0b44bc829fde8dfe4c2752 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f7469aaaedb845309056ff86fdb36244 (127.2.2.3:35781), 948a96a173de40c9895d9ce3b9af99e5 (127.2.2.1:37905)
I20260619 14:32:23.380674 7016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:23.378561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.1:37905 (local address 127.2.2.2:48339)
0619 14:32:23.378755 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:23.378772 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:23.378861 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:23.379210 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:23.379216 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:23.379232 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:23.379919 (+ 687us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:32:23.379922 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:23.379935 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:32:23.379988 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:23.380395 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:23.380398 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:23.380401 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 14:32:23.380511 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 14:32:23.380549 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260619 14:32:23.380759 7161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:23.378625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:48339 (local address 127.2.2.1:37905)
0619 14:32:23.378831 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:23.378849 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:23.378874 (+ 25us) server_negotiation.cc:407] Connection header received
0619 14:32:23.378969 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:23.378971 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:23.379064 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:23.379148 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:23.380209 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:23.380213 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:23.380216 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:32:23.380238 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:23.380267 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:32:23.380282 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:23.380402 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:23.380546 (+ 144us) server_negotiation.cc:299] Negotiation successful
0619 14:32:23.380603 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260619 14:32:23.381006 7158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:23.378651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:41403 (local address 127.2.2.3:35781)
0619 14:32:23.378817 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:23.378821 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:23.379229 (+ 408us) server_negotiation.cc:407] Connection header received
0619 14:32:23.379430 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:23.379437 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:23.379475 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:23.379552 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:23.380395 (+ 843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:23.380401 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:23.380404 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:32:23.380418 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:23.380515 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:32:23.380529 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:23.380632 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:23.380792 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 14:32:23.380837 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 14:32:23.381435 7171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:23.378576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.3:35781 (local address 127.2.2.2:41403)
0619 14:32:23.379095 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:23.379105 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:23.379257 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:23.379597 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:23.379601 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:23.379617 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:23.380226 (+ 609us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:32:23.380228 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:23.380238 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:32:23.380262 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:23.380676 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:23.380680 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:23.380684 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 14:32:23.381200 (+ 516us) client_negotiation.cc:240] Negotiation successful
0619 14:32:23.381247 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"mutex_wait_us":18,"thread_start_us":127,"threads_started":1}
I20260619 14:32:23.381518 6837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50f19b2dbbcc44b2b09af4a5dad8fc14" candidate_uuid: "b105fff72c0b44bc829fde8dfe4c2752" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948a96a173de40c9895d9ce3b9af99e5" is_pre_election: true
I20260619 14:32:23.381990 6837 raft_consensus.cc:2454] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P 948a96a173de40c9895d9ce3b9af99e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b105fff72c0b44bc829fde8dfe4c2752 in term 0.
I20260619 14:32:23.382091 7099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50f19b2dbbcc44b2b09af4a5dad8fc14" candidate_uuid: "b105fff72c0b44bc829fde8dfe4c2752" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7469aaaedb845309056ff86fdb36244" is_pre_election: true
I20260619 14:32:23.382503 7099 raft_consensus.cc:2454] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P f7469aaaedb845309056ff86fdb36244 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b105fff72c0b44bc829fde8dfe4c2752 in term 0.
I20260619 14:32:23.382637 6901 leader_election.cc:302] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P b105fff72c0b44bc829fde8dfe4c2752 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 948a96a173de40c9895d9ce3b9af99e5, b105fff72c0b44bc829fde8dfe4c2752; no voters:
I20260619 14:32:23.382941 7168 raft_consensus.cc:2781] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P b105fff72c0b44bc829fde8dfe4c2752 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:32:23.383031 7168 raft_consensus.cc:482] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P b105fff72c0b44bc829fde8dfe4c2752 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:32:23.383064 7168 raft_consensus.cc:3037] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P b105fff72c0b44bc829fde8dfe4c2752 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:23.384126 7168 raft_consensus.cc:504] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P b105fff72c0b44bc829fde8dfe4c2752 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b105fff72c0b44bc829fde8dfe4c2752" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 34179 } } peers { permanent_uuid: "f7469aaaedb845309056ff86fdb36244" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 35781 } } peers { permanent_uuid: "948a96a173de40c9895d9ce3b9af99e5" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37905 } }
I20260619 14:32:23.384517 7168 leader_election.cc:288] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P b105fff72c0b44bc829fde8dfe4c2752 [CANDIDATE]: Term 1 election: Requested vote from peers f7469aaaedb845309056ff86fdb36244 (127.2.2.3:35781), 948a96a173de40c9895d9ce3b9af99e5 (127.2.2.1:37905)
I20260619 14:32:23.384912 6837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50f19b2dbbcc44b2b09af4a5dad8fc14" candidate_uuid: "b105fff72c0b44bc829fde8dfe4c2752" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948a96a173de40c9895d9ce3b9af99e5"
I20260619 14:32:23.384938 7099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50f19b2dbbcc44b2b09af4a5dad8fc14" candidate_uuid: "b105fff72c0b44bc829fde8dfe4c2752" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7469aaaedb845309056ff86fdb36244"
I20260619 14:32:23.385066 6837 raft_consensus.cc:3037] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P 948a96a173de40c9895d9ce3b9af99e5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:23.385077 7099 raft_consensus.cc:3037] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P f7469aaaedb845309056ff86fdb36244 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:23.386211 7099 raft_consensus.cc:2454] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P f7469aaaedb845309056ff86fdb36244 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b105fff72c0b44bc829fde8dfe4c2752 in term 1.
I20260619 14:32:23.386255 6837 raft_consensus.cc:2454] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P 948a96a173de40c9895d9ce3b9af99e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b105fff72c0b44bc829fde8dfe4c2752 in term 1.
I20260619 14:32:23.386613 6901 leader_election.cc:302] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P b105fff72c0b44bc829fde8dfe4c2752 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 948a96a173de40c9895d9ce3b9af99e5, b105fff72c0b44bc829fde8dfe4c2752; no voters:
I20260619 14:32:23.386840 7168 raft_consensus.cc:2781] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P b105fff72c0b44bc829fde8dfe4c2752 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:32:23.387113 7168 raft_consensus.cc:686] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P b105fff72c0b44bc829fde8dfe4c2752 [term 1 LEADER]: Becoming Leader. State: Replica: b105fff72c0b44bc829fde8dfe4c2752, State: Running, Role: LEADER
I20260619 14:32:23.387452 7168 consensus_queue.cc:237] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P b105fff72c0b44bc829fde8dfe4c2752 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b105fff72c0b44bc829fde8dfe4c2752" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 34179 } } peers { permanent_uuid: "f7469aaaedb845309056ff86fdb36244" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 35781 } } peers { permanent_uuid: "948a96a173de40c9895d9ce3b9af99e5" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37905 } }
I20260619 14:32:23.390652 6693 catalog_manager.cc:5495] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P b105fff72c0b44bc829fde8dfe4c2752 reported cstate change: term changed from 0 to 1, leader changed from <none> to b105fff72c0b44bc829fde8dfe4c2752 (127.2.2.2). New cstate: current_term: 1 leader_uuid: "b105fff72c0b44bc829fde8dfe4c2752" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b105fff72c0b44bc829fde8dfe4c2752" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 34179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7469aaaedb845309056ff86fdb36244" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 35781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948a96a173de40c9895d9ce3b9af99e5" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37905 } health_report { overall_health: UNKNOWN } } }
I20260619 14:32:23.417471 7159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:23.415939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59538 (local address 127.2.2.2:34179)
0619 14:32:23.416071 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:23.416076 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:23.416091 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:32:23.416235 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:23.416238 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:23.416300 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:23.416398 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:23.417075 (+ 677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:23.417078 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:23.417081 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:32:23.417092 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:23.417118 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:32:23.417126 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:23.417227 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:23.417283 (+ 56us) server_negotiation.cc:299] Negotiation successful
0619 14:32:23.417325 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 14:32:23.423581 7099 raft_consensus.cc:1261] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P f7469aaaedb845309056ff86fdb36244 [term 1 FOLLOWER]: Refusing update from remote peer b105fff72c0b44bc829fde8dfe4c2752: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:32:23.423589 6837 raft_consensus.cc:1261] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P 948a96a173de40c9895d9ce3b9af99e5 [term 1 FOLLOWER]: Refusing update from remote peer b105fff72c0b44bc829fde8dfe4c2752: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:32:23.424351 7172 consensus_queue.cc:1035] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P b105fff72c0b44bc829fde8dfe4c2752 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948a96a173de40c9895d9ce3b9af99e5" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:32:23.424587 7168 consensus_queue.cc:1035] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P b105fff72c0b44bc829fde8dfe4c2752 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7469aaaedb845309056ff86fdb36244" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:32:23.434144 7177 mvcc.cc:205] Tried to move back new op lower bound from 7298578609855164416 to 7298578609717379072. Current Snapshot: MvccSnapshot[applied={T|T < 7298578609855164416}]
I20260619 14:32:23.441854 6695 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:32:23.442014 6695 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:32:23.444541 6695 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5caedc2665cc4ad3867d80443a2c9d7f: Sending DeleteTablet for 3 replicas of tablet 50f19b2dbbcc44b2b09af4a5dad8fc14
I20260619 14:32:23.445369 6948 tablet_service.cc:1514] Processing DeleteTablet for tablet 50f19b2dbbcc44b2b09af4a5dad8fc14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:32:23 UTC) from {username='slave'} at 127.0.0.1:59522
I20260619 14:32:23.445395 7079 tablet_service.cc:1514] Processing DeleteTablet for tablet 50f19b2dbbcc44b2b09af4a5dad8fc14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:32:23 UTC) from {username='slave'} at 127.0.0.1:37364
I20260619 14:32:23.445539 6817 tablet_service.cc:1514] Processing DeleteTablet for tablet 50f19b2dbbcc44b2b09af4a5dad8fc14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:32:23 UTC) from {username='slave'} at 127.0.0.1:40018
I20260619 14:32:23.445969 7185 tablet_replica.cc:330] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P b105fff72c0b44bc829fde8dfe4c2752: stopping tablet replica
I20260619 14:32:23.446183 7187 tablet_replica.cc:330] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P 948a96a173de40c9895d9ce3b9af99e5: stopping tablet replica
I20260619 14:32:23.446338 7185 raft_consensus.cc:2229] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P b105fff72c0b44bc829fde8dfe4c2752 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:32:23.446429 7187 raft_consensus.cc:2229] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P 948a96a173de40c9895d9ce3b9af99e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:32:23.446745 7185 raft_consensus.cc:2258] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P b105fff72c0b44bc829fde8dfe4c2752 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:32:23.446836 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 6755
I20260619 14:32:23.447697 7186 tablet_replica.cc:330] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P f7469aaaedb845309056ff86fdb36244: stopping tablet replica
I20260619 14:32:23.447985 7186 raft_consensus.cc:2229] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P f7469aaaedb845309056ff86fdb36244 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:32:23.448295 7186 raft_consensus.cc:2258] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P f7469aaaedb845309056ff86fdb36244 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:32:23.449720 7186 ts_tablet_manager.cc:1905] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P f7469aaaedb845309056ff86fdb36244: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:32:23.449833 7185 ts_tablet_manager.cc:1905] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P b105fff72c0b44bc829fde8dfe4c2752: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:32:23.452628 7185 ts_tablet_manager.cc:1918] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P b105fff72c0b44bc829fde8dfe4c2752: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:32:23.452731 7185 log.cc:1196] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P b105fff72c0b44bc829fde8dfe4c2752: Deleting WAL directory at /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/ts-1/wal/wals/50f19b2dbbcc44b2b09af4a5dad8fc14
I20260619 14:32:23.453075 7185 ts_tablet_manager.cc:1939] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P b105fff72c0b44bc829fde8dfe4c2752: Deleting consensus metadata
I20260619 14:32:23.453092 7186 ts_tablet_manager.cc:1918] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P f7469aaaedb845309056ff86fdb36244: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:32:23.453184 7186 log.cc:1196] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P f7469aaaedb845309056ff86fdb36244: Deleting WAL directory at /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879514993456-2056-0/minicluster-data/ts-2/wal/wals/50f19b2dbbcc44b2b09af4a5dad8fc14
I20260619 14:32:23.453509 7186 ts_tablet_manager.cc:1939] T 50f19b2dbbcc44b2b09af4a5dad8fc14 P f7469aaaedb845309056ff86fdb36244: Deleting consensus metadata
I20260619 14:32:23.454355 6681 catalog_manager.cc:4841] TS b105fff72c0b44bc829fde8dfe4c2752 (127.2.2.2:34179): tablet 50f19b2dbbcc44b2b09af4a5dad8fc14 (table test-table [id=d434fa88f55b46b3b6128caf8fd803f0]) successfully deleted
I20260619 14:32:23.454706 6680 catalog_manager.cc:4841] TS f7469aaaedb845309056ff86fdb36244 (127.2.2.3:35781): tablet 50f19b2dbbcc44b2b09af4a5dad8fc14 (table test-table [id=d434fa88f55b46b3b6128caf8fd803f0]) successfully deleted
W20260619 14:32:23.455570 6679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.2.1:37905 (error 108)
I20260619 14:32:23.456354 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 6886
I20260619 14:32:23.456676 7160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:23.456376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.1:37905 (local address 127.0.0.1:40022)
0619 14:32:23.456510 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:23.456581 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.1:37905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 14:32:23.457018 6679 catalog_manager.cc:4584] TS 948a96a173de40c9895d9ce3b9af99e5 (127.2.2.1:37905): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 50f19b2dbbcc44b2b09af4a5dad8fc14: Network error: Client connection negotiation failed: client connection to 127.2.2.1:37905: connect: Connection refused (error 111)
I20260619 14:32:23.462227 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 7017
I20260619 14:32:23.468055 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 6664
2026-06-19T14:32:23Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1061 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260619 14:32:23.489481 2056 mini_oidc.cc:152] Starting JWKS server
I20260619 14:32:23.491279 2056 webserver.cc:460] Webserver started at https://127.0.0.1:44057/ using document root <none> and password file <none>
I20260619 14:32:23.491434 2056 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 14:32:23.491667 2056 webserver.cc:460] Webserver started at http://127.0.0.1:42321/ using document root <none> and password file <none>
2026-06-19T14:32:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:32:23Z Disabled control of system clock
I20260619 14:32:23.501060 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781879514993456-2056-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781879514993456-2056-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781879514993456-2056-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:32879
--webserver_interface=127.2.2.62
--webserver_port=0
--builtin_ntp_servers=127.2.2.20:42949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.2.62:32879
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44057/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781879514993456-2056-0/testchainca.cert with env {}
W20260619 14:32:23.617985 7194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:23.618301 7194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:23.618345 7194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:23.622459 7194 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:32:23.622534 7194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:23.622550 7194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:23.622567 7194 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:32:23.622583 7194 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 14:32:23.622599 7194 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 14:32:23.622612 7194 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44057/jwks.json
I20260619 14:32:23.628285 7194 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:42949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781879514993456-2056-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781879514993456-2056-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.2.62:32879
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:32879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44057/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.2.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781879514993456-2056-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781879514993456-2056-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:32:23.629364 7194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:23.630389 7194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:32:23.638989 7199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:23.638989 7202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:23.639024 7200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:23.639518 7194 server_base.cc:1034] running on GCE node
I20260619 14:32:23.640161 7194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:23.641319 7194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:23.642541 7194 hybrid_clock.cc:648] HybridClock initialized: now 1781879543642517 us; error 36 us; skew 500 ppm
I20260619 14:32:23.644621 7194 webserver.cc:460] Webserver started at http://127.2.2.62:37621/ using document root <none> and password file <none>
I20260619 14:32:23.645226 7194 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:23.645318 7194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:23.645552 7194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:32:23.647657 7194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781879514993456-2056-0/minicluster-data/master-0/data/instance:
uuid: "afe3153d1aeb45a7a15483d3207fce3f"
format_stamp: "Formatted at 2026-06-19 14:32:23 on dist-test-slave-70rm"
server_key: "d0991a0d1ee921cd914a850dbf739607"
server_key_iv: "cb3d06505ece8627dfdb73516f022de3"
server_key_version: "encryptionkey@0"
I20260619 14:32:23.648284 7194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781879514993456-2056-0/minicluster-data/master-0/wal/instance:
uuid: "afe3153d1aeb45a7a15483d3207fce3f"
format_stamp: "Formatted at 2026-06-19 14:32:23 on dist-test-slave-70rm"
server_key: "d0991a0d1ee921cd914a850dbf739607"
server_key_iv: "cb3d06505ece8627dfdb73516f022de3"
server_key_version: "encryptionkey@0"
I20260619 14:32:23.652120 7194 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:32:23.654605 7208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:23.655851 7194 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:32:23.655998 7194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781879514993456-2056-0/minicluster-data/master-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781879514993456-2056-0/minicluster-data/master-0/wal
uuid: "afe3153d1aeb45a7a15483d3207fce3f"
format_stamp: "Formatted at 2026-06-19 14:32:23 on dist-test-slave-70rm"
server_key: "d0991a0d1ee921cd914a850dbf739607"
server_key_iv: "cb3d06505ece8627dfdb73516f022de3"
server_key_version: "encryptionkey@0"
I20260619 14:32:23.656121 7194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781879514993456-2056-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781879514993456-2056-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781879514993456-2056-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:23.671813 7194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:23.672583 7194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:32:23.672797 7194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:23.680063 7194 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.62:32879
I20260619 14:32:23.680068 7260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.62:32879 every 8 connection(s)
I20260619 14:32:23.681073 7194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
I20260619 14:32:23.684363 7261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:23.688745 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 7194
I20260619 14:32:23.688922 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781879514993456-2056-0/minicluster-data/master-0/wal/instance
I20260619 14:32:23.689254 2056 external_mini_cluster.cc:1428] Setting key fab3302734c30be7bb60af279559bc2d
I20260619 14:32:23.690179 7261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afe3153d1aeb45a7a15483d3207fce3f: Bootstrap starting.
I20260619 14:32:23.692764 7261 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P afe3153d1aeb45a7a15483d3207fce3f: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:23.693688 7261 log.cc:824] T 00000000000000000000000000000000 P afe3153d1aeb45a7a15483d3207fce3f: Log is configured to *not* fsync() on all Append() calls
I20260619 14:32:23.696106 7261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afe3153d1aeb45a7a15483d3207fce3f: No bootstrap required, opened a new log
I20260619 14:32:23.699611 7264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:23.690713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35682 (local address 127.2.2.62:32879)
0619 14:32:23.691532 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:23.691547 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:23.691598 (+ 51us) server_negotiation.cc:407] Connection header received
0619 14:32:23.692270 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:23.692288 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:23.692719 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:23.693079 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:23.693459 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:23.694740 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:23.696033 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:23.696617 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:23.696751 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:23.697623 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:23.697628 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:23.697641 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:32:23.698050 (+ 409us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:23.698109 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:32:23.698180 (+ 71us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:23.698264 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:23.698465 (+ 201us) server_negotiation.cc:299] Negotiation successful
0619 14:32:23.698803 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":124,"threads_started":1}
I20260619 14:32:23.700443 7261 raft_consensus.cc:348] T 00000000000000000000000000000000 P afe3153d1aeb45a7a15483d3207fce3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afe3153d1aeb45a7a15483d3207fce3f" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 32879 } }
I20260619 14:32:23.700736 7261 raft_consensus.cc:374] T 00000000000000000000000000000000 P afe3153d1aeb45a7a15483d3207fce3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:23.700829 7261 raft_consensus.cc:729] T 00000000000000000000000000000000 P afe3153d1aeb45a7a15483d3207fce3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afe3153d1aeb45a7a15483d3207fce3f, State: Initialized, Role: FOLLOWER
I20260619 14:32:23.701299 7261 consensus_queue.cc:260] T 00000000000000000000000000000000 P afe3153d1aeb45a7a15483d3207fce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afe3153d1aeb45a7a15483d3207fce3f" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 32879 } }
I20260619 14:32:23.701419 7261 raft_consensus.cc:388] T 00000000000000000000000000000000 P afe3153d1aeb45a7a15483d3207fce3f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:32:23.701499 7261 raft_consensus.cc:482] T 00000000000000000000000000000000 P afe3153d1aeb45a7a15483d3207fce3f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:32:23.701612 7261 raft_consensus.cc:3037] T 00000000000000000000000000000000 P afe3153d1aeb45a7a15483d3207fce3f [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:23.702759 2056 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:32:23.702785 7261 raft_consensus.cc:504] T 00000000000000000000000000000000 P afe3153d1aeb45a7a15483d3207fce3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afe3153d1aeb45a7a15483d3207fce3f" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 32879 } }
I20260619 14:32:23.703142 7261 leader_election.cc:302] T 00000000000000000000000000000000 P afe3153d1aeb45a7a15483d3207fce3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: afe3153d1aeb45a7a15483d3207fce3f; no voters:
I20260619 14:32:23.703495 7261 leader_election.cc:288] T 00000000000000000000000000000000 P afe3153d1aeb45a7a15483d3207fce3f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:32:23.703572 7266 raft_consensus.cc:2781] T 00000000000000000000000000000000 P afe3153d1aeb45a7a15483d3207fce3f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:32:23.703785 7266 raft_consensus.cc:686] T 00000000000000000000000000000000 P afe3153d1aeb45a7a15483d3207fce3f [term 1 LEADER]: Becoming Leader. State: Replica: afe3153d1aeb45a7a15483d3207fce3f, State: Running, Role: LEADER
I20260619 14:32:23.704115 7266 consensus_queue.cc:237] T 00000000000000000000000000000000 P afe3153d1aeb45a7a15483d3207fce3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afe3153d1aeb45a7a15483d3207fce3f" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 32879 } }
I20260619 14:32:23.704846 7261 sys_catalog.cc:564] T 00000000000000000000000000000000 P afe3153d1aeb45a7a15483d3207fce3f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:32:23.705790 7268 sys_catalog.cc:455] T 00000000000000000000000000000000 P afe3153d1aeb45a7a15483d3207fce3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "afe3153d1aeb45a7a15483d3207fce3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afe3153d1aeb45a7a15483d3207fce3f" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 32879 } } }
I20260619 14:32:23.705924 7268 sys_catalog.cc:458] T 00000000000000000000000000000000 P afe3153d1aeb45a7a15483d3207fce3f [sys.catalog]: This master's current role is: LEADER
I20260619 14:32:23.706249 7267 sys_catalog.cc:455] T 00000000000000000000000000000000 P afe3153d1aeb45a7a15483d3207fce3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader afe3153d1aeb45a7a15483d3207fce3f. Latest consensus state: current_term: 1 leader_uuid: "afe3153d1aeb45a7a15483d3207fce3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afe3153d1aeb45a7a15483d3207fce3f" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 32879 } } }
I20260619 14:32:23.706346 7267 sys_catalog.cc:458] T 00000000000000000000000000000000 P afe3153d1aeb45a7a15483d3207fce3f [sys.catalog]: This master's current role is: LEADER
I20260619 14:32:23.706583 7275 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:32:23.709266 7275 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:32:23.715189 7275 catalog_manager.cc:1347] Generated new cluster ID: 16b0ceaacccd4fe582038846d7d28bde
I20260619 14:32:23.715265 7275 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:32:23.730453 7275 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:32:23.731740 7275 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:32:23.748912 7275 catalog_manager.cc:5868] T 00000000000000000000000000000000 P afe3153d1aeb45a7a15483d3207fce3f: Generated new TSK 0
I20260619 14:32:23.749840 7275 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 14:32:23.783182 7291 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260619 14:32:23.783331 7291 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 14:32:23.783963 7264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:23.766552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35690 (local address 127.2.2.62:32879)
0619 14:32:23.766686 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:23.766691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:23.766868 (+ 177us) server_negotiation.cc:407] Connection header received
0619 14:32:23.767073 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:23.767076 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:23.767136 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:23.767256 (+ 120us) server_negotiation.cc:226] Negotiated authn=JWT
0619 14:32:23.767501 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:23.768339 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:23.769182 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:23.769545 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:23.769649 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:23.769973 (+ 324us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 14:32:23.783507 (+ 13534us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44057/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 14:32:23.783811 (+ 304us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35690: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44057/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":57}
W20260619 14:32:23.784053 7264 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35690: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44057/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260619 14:32:23.785132 7290 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:32:23.766441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:32879 (local address 127.0.0.1:35690)
0619 14:32:23.766778 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:23.766791 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:23.766904 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:23.767279 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:23.767281 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:23.767288 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0619 14:32:23.767392 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:23.767396 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:23.768452 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:23.768455 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:23.769069 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:23.769076 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:23.769644 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:23.769663 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:23.769733 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:23.769778 (+ 45us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 14:32:23.783779 (+ 14001us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44057/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 14:32:23.783987 (+ 208us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.2.62:32879: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44057/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":115,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.2.2.62:32879: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44057/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260619 14:32:23.786310 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 7194
2026-06-19T14:32:23Z chronyd exiting
I20260619 14:32:23.823074 2056 test_util.cc:175] -----------------------------------------------
I20260619 14:32:23.823186 2056 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781879514993456-2056-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (335 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260619 14:32:23.824249 2056 mini_oidc.cc:152] Starting JWKS server
I20260619 14:32:23.825111 2056 webserver.cc:460] Webserver started at https://127.0.0.1:45707/ using document root <none> and password file <none>
I20260619 14:32:23.825246 2056 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 14:32:23.825539 2056 webserver.cc:460] Webserver started at http://127.0.0.1:44495/ using document root <none> and password file <none>
2026-06-19T14:32:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:32:23Z Disabled control of system clock
I20260619 14:32:23.834240 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781879514993456-2056-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781879514993456-2056-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781879514993456-2056-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:40833
--webserver_interface=127.2.2.62
--webserver_port=0
--builtin_ntp_servers=127.2.2.20:43757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.2.62:40833
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45707/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781879514993456-2056-0/testchainca.cert with env {}
W20260619 14:32:23.946612 7297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:23.947055 7297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:23.947151 7297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:23.951376 7297 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:32:23.951442 7297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:23.951458 7297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:23.951475 7297 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:32:23.951491 7297 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 14:32:23.951535 7297 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 14:32:23.951565 7297 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45707/jwks.json
I20260619 14:32:23.957063 7297 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:43757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781879514993456-2056-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781879514993456-2056-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.2.62:40833
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:40833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45707/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.2.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781879514993456-2056-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781879514993456-2056-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:32:23.958258 7297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:23.959408 7297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:32:23.968007 7305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:23.968020 7303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:23.968122 7297 server_base.cc:1034] running on GCE node
W20260619 14:32:23.968009 7302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:23.968904 7297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:23.970077 7297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:23.971287 7297 hybrid_clock.cc:648] HybridClock initialized: now 1781879543971273 us; error 35 us; skew 500 ppm
I20260619 14:32:23.973126 7297 webserver.cc:460] Webserver started at http://127.2.2.62:45627/ using document root <none> and password file <none>
I20260619 14:32:23.973711 7297 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:23.973773 7297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:23.973978 7297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:32:23.976068 7297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781879514993456-2056-0/minicluster-data/master-0/data/instance:
uuid: "bf4c609ce2f9488c97471217f6f8c27f"
format_stamp: "Formatted at 2026-06-19 14:32:23 on dist-test-slave-70rm"
server_key: "dc1dc10aaca83363caf295dd464a371c"
server_key_iv: "d602c5bc618dc4cb683db7b73dc7ca42"
server_key_version: "encryptionkey@0"
I20260619 14:32:23.976634 7297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781879514993456-2056-0/minicluster-data/master-0/wal/instance:
uuid: "bf4c609ce2f9488c97471217f6f8c27f"
format_stamp: "Formatted at 2026-06-19 14:32:23 on dist-test-slave-70rm"
server_key: "dc1dc10aaca83363caf295dd464a371c"
server_key_iv: "d602c5bc618dc4cb683db7b73dc7ca42"
server_key_version: "encryptionkey@0"
I20260619 14:32:23.980372 7297 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.005s
I20260619 14:32:23.982874 7311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:23.984148 7297 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:32:23.984292 7297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781879514993456-2056-0/minicluster-data/master-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781879514993456-2056-0/minicluster-data/master-0/wal
uuid: "bf4c609ce2f9488c97471217f6f8c27f"
format_stamp: "Formatted at 2026-06-19 14:32:23 on dist-test-slave-70rm"
server_key: "dc1dc10aaca83363caf295dd464a371c"
server_key_iv: "d602c5bc618dc4cb683db7b73dc7ca42"
server_key_version: "encryptionkey@0"
I20260619 14:32:23.984411 7297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781879514993456-2056-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781879514993456-2056-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781879514993456-2056-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:24.023365 7297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:24.024127 7297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:32:24.024348 7297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:24.031682 7297 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.62:40833
I20260619 14:32:24.031692 7363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.62:40833 every 8 connection(s)
I20260619 14:32:24.032734 7297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
I20260619 14:32:24.036015 7364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:24.040616 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 7297
I20260619 14:32:24.040772 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781879514993456-2056-0/minicluster-data/master-0/wal/instance
I20260619 14:32:24.041049 2056 external_mini_cluster.cc:1428] Setting key f637eb2086821949e0d8bff76c601d36
I20260619 14:32:24.042081 7364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf4c609ce2f9488c97471217f6f8c27f: Bootstrap starting.
I20260619 14:32:24.044478 7364 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf4c609ce2f9488c97471217f6f8c27f: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:24.045238 7364 log.cc:824] T 00000000000000000000000000000000 P bf4c609ce2f9488c97471217f6f8c27f: Log is configured to *not* fsync() on all Append() calls
I20260619 14:32:24.047247 7364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf4c609ce2f9488c97471217f6f8c27f: No bootstrap required, opened a new log
I20260619 14:32:24.050554 7367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:24.042465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38010 (local address 127.2.2.62:40833)
0619 14:32:24.043111 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:24.043125 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:24.043161 (+ 36us) server_negotiation.cc:407] Connection header received
0619 14:32:24.043805 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:24.043822 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:24.044230 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:24.044579 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:24.045019 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:24.046032 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:24.046965 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:24.047499 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:24.047684 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:24.048646 (+ 962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:24.048654 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:24.048674 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:32:24.049069 (+ 395us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:24.049126 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:32:24.049147 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:24.049273 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:24.049444 (+ 171us) server_negotiation.cc:299] Negotiation successful
0619 14:32:24.049719 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":138,"threads_started":1}
I20260619 14:32:24.051157 7364 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf4c609ce2f9488c97471217f6f8c27f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf4c609ce2f9488c97471217f6f8c27f" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 40833 } }
I20260619 14:32:24.051415 7364 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf4c609ce2f9488c97471217f6f8c27f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:24.051487 7364 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf4c609ce2f9488c97471217f6f8c27f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf4c609ce2f9488c97471217f6f8c27f, State: Initialized, Role: FOLLOWER
I20260619 14:32:24.051920 7364 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf4c609ce2f9488c97471217f6f8c27f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf4c609ce2f9488c97471217f6f8c27f" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 40833 } }
I20260619 14:32:24.052044 7364 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf4c609ce2f9488c97471217f6f8c27f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:32:24.052127 7364 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf4c609ce2f9488c97471217f6f8c27f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:32:24.052235 7364 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf4c609ce2f9488c97471217f6f8c27f [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:24.053393 7364 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf4c609ce2f9488c97471217f6f8c27f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf4c609ce2f9488c97471217f6f8c27f" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 40833 } }
I20260619 14:32:24.053521 2056 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:32:24.053783 7364 leader_election.cc:302] T 00000000000000000000000000000000 P bf4c609ce2f9488c97471217f6f8c27f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf4c609ce2f9488c97471217f6f8c27f; no voters:
I20260619 14:32:24.054117 7364 leader_election.cc:288] T 00000000000000000000000000000000 P bf4c609ce2f9488c97471217f6f8c27f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:32:24.054235 7369 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf4c609ce2f9488c97471217f6f8c27f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:32:24.054440 7369 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf4c609ce2f9488c97471217f6f8c27f [term 1 LEADER]: Becoming Leader. State: Replica: bf4c609ce2f9488c97471217f6f8c27f, State: Running, Role: LEADER
I20260619 14:32:24.054755 7369 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf4c609ce2f9488c97471217f6f8c27f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf4c609ce2f9488c97471217f6f8c27f" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 40833 } }
I20260619 14:32:24.055552 7364 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf4c609ce2f9488c97471217f6f8c27f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:32:24.056488 7371 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf4c609ce2f9488c97471217f6f8c27f [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf4c609ce2f9488c97471217f6f8c27f. Latest consensus state: current_term: 1 leader_uuid: "bf4c609ce2f9488c97471217f6f8c27f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf4c609ce2f9488c97471217f6f8c27f" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 40833 } } }
I20260619 14:32:24.056610 7371 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf4c609ce2f9488c97471217f6f8c27f [sys.catalog]: This master's current role is: LEADER
I20260619 14:32:24.056449 7370 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf4c609ce2f9488c97471217f6f8c27f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf4c609ce2f9488c97471217f6f8c27f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf4c609ce2f9488c97471217f6f8c27f" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 40833 } } }
I20260619 14:32:24.056811 7370 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf4c609ce2f9488c97471217f6f8c27f [sys.catalog]: This master's current role is: LEADER
I20260619 14:32:24.057042 7378 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:32:24.059847 7378 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:32:24.065680 7378 catalog_manager.cc:1347] Generated new cluster ID: 75c3f353bccc4b4aa2151f2f0a77aea2
I20260619 14:32:24.065766 7378 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:32:24.071002 7378 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:32:24.072247 7378 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:32:24.079123 7378 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf4c609ce2f9488c97471217f6f8c27f: Generated new TSK 0
I20260619 14:32:24.079952 7378 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 14:32:24.111245 7394 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260619 14:32:24.111344 7394 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 14:32:24.112149 7367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:24.096324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38026 (local address 127.2.2.62:40833)
0619 14:32:24.096454 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:24.096460 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:24.096642 (+ 182us) server_negotiation.cc:407] Connection header received
0619 14:32:24.096869 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:24.096871 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:24.096920 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:24.097034 (+ 114us) server_negotiation.cc:226] Negotiated authn=JWT
0619 14:32:24.097288 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:24.097836 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:24.098786 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:24.099017 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:24.099131 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:24.099445 (+ 314us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 14:32:24.111679 (+ 12234us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45707/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 14:32:24.111986 (+ 307us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38026: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45707/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":45}
W20260619 14:32:24.112231 7393 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:32:24.096242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:40833 (local address 127.0.0.1:38026)
0619 14:32:24.096550 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:24.096562 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:24.096710 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:24.097044 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:24.097047 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:24.097054 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0619 14:32:24.097182 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:24.097187 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:24.097951 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:24.097955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:24.098657 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:24.098665 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:24.099120 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:24.099141 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:24.099236 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:24.099269 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 14:32:24.111894 (+ 12625us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45707/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 14:32:24.112064 (+ 170us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.2.62:40833: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45707/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":117,"threads_started":1}
W20260619 14:32:24.112239 7367 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38026: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45707/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260619 14:32:24.112654 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 7297
2026-06-19T14:32:24Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (327 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260619 14:32:24.151736 2056 mini_oidc.cc:152] Starting JWKS server
I20260619 14:32:24.152596 2056 webserver.cc:460] Webserver started at https://127.0.0.1:42007/ using document root <none> and password file <none>
I20260619 14:32:24.152715 2056 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 14:32:24.153004 2056 webserver.cc:460] Webserver started at http://127.0.0.1:40499/ using document root <none> and password file <none>
2026-06-19T14:32:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:32:24Z Disabled control of system clock
I20260619 14:32:24.161634 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781879514993456-2056-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781879514993456-2056-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781879514993456-2056-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:38541
--webserver_interface=127.2.2.62
--webserver_port=0
--builtin_ntp_servers=127.2.2.20:37603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.2.62:38541
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42007/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 14:32:24.272014 7400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:24.272341 7400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:24.272418 7400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:24.276621 7400 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:32:24.276729 7400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:24.276759 7400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:24.276803 7400 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:32:24.276839 7400 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 14:32:24.276876 7400 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 14:32:24.276908 7400 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42007/jwks.json
I20260619 14:32:24.281810 7400 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:37603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781879514993456-2056-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781879514993456-2056-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.2.62:38541
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:38541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42007/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.2.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781879514993456-2056-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:32:24.283068 7400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:24.284490 7400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:32:24.293288 7406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:24.293316 7405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:24.293346 7400 server_base.cc:1034] running on GCE node
W20260619 14:32:24.293288 7408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:24.294106 7400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:24.295329 7400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:24.296514 7400 hybrid_clock.cc:648] HybridClock initialized: now 1781879544296478 us; error 61 us; skew 500 ppm
I20260619 14:32:24.298517 7400 webserver.cc:460] Webserver started at http://127.2.2.62:41673/ using document root <none> and password file <none>
I20260619 14:32:24.299130 7400 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:24.299232 7400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:24.299465 7400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:32:24.301568 7400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781879514993456-2056-0/minicluster-data/master-0/data/instance:
uuid: "4758609849274def8beeba455dadfed3"
format_stamp: "Formatted at 2026-06-19 14:32:24 on dist-test-slave-70rm"
server_key: "b7915de69ad3b86f889b0b47cb596338"
server_key_iv: "2be244fb0f7107334d0e09616b9087ac"
server_key_version: "encryptionkey@0"
I20260619 14:32:24.302215 7400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781879514993456-2056-0/minicluster-data/master-0/wal/instance:
uuid: "4758609849274def8beeba455dadfed3"
format_stamp: "Formatted at 2026-06-19 14:32:24 on dist-test-slave-70rm"
server_key: "b7915de69ad3b86f889b0b47cb596338"
server_key_iv: "2be244fb0f7107334d0e09616b9087ac"
server_key_version: "encryptionkey@0"
I20260619 14:32:24.305903 7400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 14:32:24.308440 7414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:24.309638 7400 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:32:24.309785 7400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781879514993456-2056-0/minicluster-data/master-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781879514993456-2056-0/minicluster-data/master-0/wal
uuid: "4758609849274def8beeba455dadfed3"
format_stamp: "Formatted at 2026-06-19 14:32:24 on dist-test-slave-70rm"
server_key: "b7915de69ad3b86f889b0b47cb596338"
server_key_iv: "2be244fb0f7107334d0e09616b9087ac"
server_key_version: "encryptionkey@0"
I20260619 14:32:24.309901 7400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781879514993456-2056-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781879514993456-2056-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781879514993456-2056-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:24.321384 7400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:24.322090 7400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:32:24.322331 7400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:24.329262 7400 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.62:38541
I20260619 14:32:24.329277 7466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.62:38541 every 8 connection(s)
I20260619 14:32:24.330293 7400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
I20260619 14:32:24.333580 7467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:24.337777 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 7400
I20260619 14:32:24.337884 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781879514993456-2056-0/minicluster-data/master-0/wal/instance
I20260619 14:32:24.338205 2056 external_mini_cluster.cc:1428] Setting key 9dbb77ccb0f99245a2b1216de1734912
I20260619 14:32:24.339401 7467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4758609849274def8beeba455dadfed3: Bootstrap starting.
I20260619 14:32:24.341624 7467 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4758609849274def8beeba455dadfed3: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:24.342516 7467 log.cc:824] T 00000000000000000000000000000000 P 4758609849274def8beeba455dadfed3: Log is configured to *not* fsync() on all Append() calls
I20260619 14:32:24.344645 7467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4758609849274def8beeba455dadfed3: No bootstrap required, opened a new log
I20260619 14:32:24.348462 7470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:24.339622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35444 (local address 127.2.2.62:38541)
0619 14:32:24.340254 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:24.340269 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:24.340304 (+ 35us) server_negotiation.cc:407] Connection header received
0619 14:32:24.340925 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:24.340941 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:24.341344 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:24.341619 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:24.342039 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:24.343110 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:24.344005 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:24.344697 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:24.344857 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:24.345799 (+ 942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:24.345806 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:24.345823 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:32:24.346389 (+ 566us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:24.346460 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:32:24.346486 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:24.346583 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:24.346741 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 14:32:24.347019 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":143,"threads_started":1}
I20260619 14:32:24.348677 7467 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4758609849274def8beeba455dadfed3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4758609849274def8beeba455dadfed3" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 38541 } }
I20260619 14:32:24.348966 7467 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4758609849274def8beeba455dadfed3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:24.349015 7467 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4758609849274def8beeba455dadfed3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4758609849274def8beeba455dadfed3, State: Initialized, Role: FOLLOWER
I20260619 14:32:24.349472 7467 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4758609849274def8beeba455dadfed3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4758609849274def8beeba455dadfed3" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 38541 } }
I20260619 14:32:24.349658 7467 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4758609849274def8beeba455dadfed3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:32:24.349753 7467 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4758609849274def8beeba455dadfed3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:32:24.349854 7467 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4758609849274def8beeba455dadfed3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:24.351092 7467 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4758609849274def8beeba455dadfed3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4758609849274def8beeba455dadfed3" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 38541 } }
I20260619 14:32:24.351469 7467 leader_election.cc:302] T 00000000000000000000000000000000 P 4758609849274def8beeba455dadfed3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4758609849274def8beeba455dadfed3; no voters:
I20260619 14:32:24.351848 7467 leader_election.cc:288] T 00000000000000000000000000000000 P 4758609849274def8beeba455dadfed3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:32:24.351850 2056 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:32:24.351965 7472 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4758609849274def8beeba455dadfed3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:32:24.352159 7472 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4758609849274def8beeba455dadfed3 [term 1 LEADER]: Becoming Leader. State: Replica: 4758609849274def8beeba455dadfed3, State: Running, Role: LEADER
I20260619 14:32:24.352543 7472 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4758609849274def8beeba455dadfed3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4758609849274def8beeba455dadfed3" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 38541 } }
I20260619 14:32:24.353157 7467 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4758609849274def8beeba455dadfed3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:32:24.354529 7473 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4758609849274def8beeba455dadfed3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4758609849274def8beeba455dadfed3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4758609849274def8beeba455dadfed3" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 38541 } } }
I20260619 14:32:24.354537 7474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4758609849274def8beeba455dadfed3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4758609849274def8beeba455dadfed3. Latest consensus state: current_term: 1 leader_uuid: "4758609849274def8beeba455dadfed3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4758609849274def8beeba455dadfed3" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 38541 } } }
I20260619 14:32:24.354730 7474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4758609849274def8beeba455dadfed3 [sys.catalog]: This master's current role is: LEADER
I20260619 14:32:24.354753 7473 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4758609849274def8beeba455dadfed3 [sys.catalog]: This master's current role is: LEADER
I20260619 14:32:24.355455 7481 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:32:24.358214 7481 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:32:24.363549 7481 catalog_manager.cc:1347] Generated new cluster ID: 919265b9255d42f9974411de04635b05
I20260619 14:32:24.363626 7481 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:32:24.370167 7481 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:32:24.371068 7481 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:32:24.375849 7481 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4758609849274def8beeba455dadfed3: Generated new TSK 0
I20260619 14:32:24.376667 7481 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:32:24.396586 7470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:24.390914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35456 (local address 127.2.2.62:38541)
0619 14:32:24.391066 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:24.391071 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:24.391322 (+ 251us) server_negotiation.cc:407] Connection header received
0619 14:32:24.391485 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:24.391489 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:24.391552 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:24.391689 (+ 137us) server_negotiation.cc:226] Negotiated authn=JWT
0619 14:32:24.391931 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:24.392784 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:24.393713 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:24.394059 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:24.394191 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:24.394572 (+ 381us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 14:32:24.396176 (+ 1604us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42007/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 14:32:24.396423 (+ 247us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35456: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42007/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":59}
W20260619 14:32:24.396667 7470 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35456: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42007/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260619 14:32:24.398250 7496 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:32:24.390830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:38541 (local address 127.0.0.1:35456)
0619 14:32:24.391221 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:24.391234 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:24.391336 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:24.391687 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:24.391689 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:24.391695 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0619 14:32:24.391822 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:24.391827 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:24.392914 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:24.392917 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:24.393587 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:24.393596 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:24.394215 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:24.394236 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:24.394306 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:24.394331 (+ 25us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 14:32:24.396372 (+ 2041us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42007/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 14:32:24.398027 (+ 1655us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.2.62:38541: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42007/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":1442,"thread_start_us":114,"threads_started":1}
I20260619 14:32:24.398617 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 7400
E20260619 14:32:24.407495 7497 webserver.cc:533] Webserver: sslize failed:
E20260619 14:32:24.407577 7497 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-19T14:32:24Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (285 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-19T14:32:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:32:24Z Disabled control of system clock
I20260619 14:32:24.445827 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781879514993456-2056-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781879514993456-2056-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781879514993456-2056-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:36025
--webserver_interface=127.2.2.62
--webserver_port=0
--builtin_ntp_servers=127.2.2.20:34017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.2.62:36025
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781879514993456-2056-0/testchainca.cert with env {}
W20260619 14:32:24.559186 7501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:24.559561 7501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:24.559649 7501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:24.563925 7501 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:32:24.564004 7501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:24.564021 7501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:24.564039 7501 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:32:24.564054 7501 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 14:32:24.564116 7501 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 14:32:24.564146 7501 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260619 14:32:24.569892 7501 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:34017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781879514993456-2056-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781879514993456-2056-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.2.62:36025
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:36025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.2.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781879514993456-2056-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781879514993456-2056-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:32:24.571100 7501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:24.572160 7501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:32:24.581173 7507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:24.581281 7506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:24.581269 7501 server_base.cc:1034] running on GCE node
W20260619 14:32:24.581202 7509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:24.582159 7501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:24.583381 7501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:24.584609 7501 hybrid_clock.cc:648] HybridClock initialized: now 1781879544584598 us; error 36 us; skew 500 ppm
I20260619 14:32:24.586848 7501 webserver.cc:460] Webserver started at http://127.2.2.62:36467/ using document root <none> and password file <none>
I20260619 14:32:24.587464 7501 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:24.587570 7501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:24.587810 7501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:32:24.589968 7501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781879514993456-2056-0/minicluster-data/master-0/data/instance:
uuid: "e9f3e2dbae504eb59bbeca8aabf017fc"
format_stamp: "Formatted at 2026-06-19 14:32:24 on dist-test-slave-70rm"
server_key: "5d1a9caca15aa33be9d89495c30b2394"
server_key_iv: "0d385cee4d3a689c244e580783494769"
server_key_version: "encryptionkey@0"
I20260619 14:32:24.590619 7501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781879514993456-2056-0/minicluster-data/master-0/wal/instance:
uuid: "e9f3e2dbae504eb59bbeca8aabf017fc"
format_stamp: "Formatted at 2026-06-19 14:32:24 on dist-test-slave-70rm"
server_key: "5d1a9caca15aa33be9d89495c30b2394"
server_key_iv: "0d385cee4d3a689c244e580783494769"
server_key_version: "encryptionkey@0"
I20260619 14:32:24.594555 7501 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:32:24.597139 7515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:24.598585 7501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:32:24.598737 7501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781879514993456-2056-0/minicluster-data/master-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781879514993456-2056-0/minicluster-data/master-0/wal
uuid: "e9f3e2dbae504eb59bbeca8aabf017fc"
format_stamp: "Formatted at 2026-06-19 14:32:24 on dist-test-slave-70rm"
server_key: "5d1a9caca15aa33be9d89495c30b2394"
server_key_iv: "0d385cee4d3a689c244e580783494769"
server_key_version: "encryptionkey@0"
I20260619 14:32:24.598862 7501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781879514993456-2056-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781879514993456-2056-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781879514993456-2056-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:24.622699 7501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:24.623456 7501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:32:24.623672 7501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:24.631368 7501 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.62:36025
I20260619 14:32:24.631407 7567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.62:36025 every 8 connection(s)
I20260619 14:32:24.632390 7501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
I20260619 14:32:24.635682 7568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:24.641314 7568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9f3e2dbae504eb59bbeca8aabf017fc: Bootstrap starting.
I20260619 14:32:24.641813 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 7501
I20260619 14:32:24.641901 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781879514993456-2056-0/minicluster-data/master-0/wal/instance
I20260619 14:32:24.642269 2056 external_mini_cluster.cc:1428] Setting key 7730b6868b708911c3f2bebfe92109be
I20260619 14:32:24.644078 7568 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9f3e2dbae504eb59bbeca8aabf017fc: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:24.644858 7568 log.cc:824] T 00000000000000000000000000000000 P e9f3e2dbae504eb59bbeca8aabf017fc: Log is configured to *not* fsync() on all Append() calls
I20260619 14:32:24.647073 7568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9f3e2dbae504eb59bbeca8aabf017fc: No bootstrap required, opened a new log
I20260619 14:32:24.651520 7568 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9f3e2dbae504eb59bbeca8aabf017fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9f3e2dbae504eb59bbeca8aabf017fc" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 36025 } }
I20260619 14:32:24.651778 7568 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9f3e2dbae504eb59bbeca8aabf017fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:24.651867 7568 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9f3e2dbae504eb59bbeca8aabf017fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9f3e2dbae504eb59bbeca8aabf017fc, State: Initialized, Role: FOLLOWER
I20260619 14:32:24.651892 7571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:24.643617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60642 (local address 127.2.2.62:36025)
0619 14:32:24.644299 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:24.644314 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:24.644352 (+ 38us) server_negotiation.cc:407] Connection header received
0619 14:32:24.645012 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:24.645035 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:24.645561 (+ 526us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:24.645906 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:24.646336 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:24.647348 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:24.648535 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:24.648994 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:24.649123 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:24.649951 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:24.649956 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:24.649969 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:32:24.650448 (+ 479us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:24.650502 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:32:24.650522 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:24.650598 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:24.650793 (+ 195us) server_negotiation.cc:299] Negotiation successful
0619 14:32:24.651087 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":136,"threads_started":1}
I20260619 14:32:24.652415 7568 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9f3e2dbae504eb59bbeca8aabf017fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9f3e2dbae504eb59bbeca8aabf017fc" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 36025 } }
I20260619 14:32:24.652575 7568 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9f3e2dbae504eb59bbeca8aabf017fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:32:24.652634 7568 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9f3e2dbae504eb59bbeca8aabf017fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:32:24.652725 7568 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9f3e2dbae504eb59bbeca8aabf017fc [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:24.653736 7568 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9f3e2dbae504eb59bbeca8aabf017fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9f3e2dbae504eb59bbeca8aabf017fc" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 36025 } }
I20260619 14:32:24.654069 7568 leader_election.cc:302] T 00000000000000000000000000000000 P e9f3e2dbae504eb59bbeca8aabf017fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9f3e2dbae504eb59bbeca8aabf017fc; no voters:
I20260619 14:32:24.654471 7568 leader_election.cc:288] T 00000000000000000000000000000000 P e9f3e2dbae504eb59bbeca8aabf017fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:32:24.654577 7573 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9f3e2dbae504eb59bbeca8aabf017fc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:32:24.654793 2056 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:32:24.654780 7573 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9f3e2dbae504eb59bbeca8aabf017fc [term 1 LEADER]: Becoming Leader. State: Replica: e9f3e2dbae504eb59bbeca8aabf017fc, State: Running, Role: LEADER
I20260619 14:32:24.655071 7573 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9f3e2dbae504eb59bbeca8aabf017fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9f3e2dbae504eb59bbeca8aabf017fc" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 36025 } }
I20260619 14:32:24.655707 7568 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9f3e2dbae504eb59bbeca8aabf017fc [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:32:24.656512 7574 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9f3e2dbae504eb59bbeca8aabf017fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9f3e2dbae504eb59bbeca8aabf017fc. Latest consensus state: current_term: 1 leader_uuid: "e9f3e2dbae504eb59bbeca8aabf017fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9f3e2dbae504eb59bbeca8aabf017fc" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 36025 } } }
I20260619 14:32:24.656656 7574 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9f3e2dbae504eb59bbeca8aabf017fc [sys.catalog]: This master's current role is: LEADER
I20260619 14:32:24.657280 7575 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9f3e2dbae504eb59bbeca8aabf017fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9f3e2dbae504eb59bbeca8aabf017fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9f3e2dbae504eb59bbeca8aabf017fc" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 36025 } } }
I20260619 14:32:24.657399 7575 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9f3e2dbae504eb59bbeca8aabf017fc [sys.catalog]: This master's current role is: LEADER
I20260619 14:32:24.657969 7582 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:32:24.661032 7582 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:32:24.666409 7582 catalog_manager.cc:1347] Generated new cluster ID: 163c3a87a92e4316b3816631d1615a45
I20260619 14:32:24.666494 7582 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:32:24.680362 7582 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:32:24.681809 7582 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:32:24.688319 7582 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9f3e2dbae504eb59bbeca8aabf017fc: Generated new TSK 0
I20260619 14:32:24.689085 7582 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:32:25.156512 7571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:24.699204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60654 (local address 127.2.2.62:36025)
0619 14:32:24.699345 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:24.699350 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:24.699501 (+ 151us) server_negotiation.cc:407] Connection header received
0619 14:32:24.699648 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:24.699652 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:24.699711 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:24.699849 (+ 138us) server_negotiation.cc:226] Negotiated authn=JWT
0619 14:32:24.700034 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:24.700888 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:24.701863 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:24.702253 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:24.702357 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:24.702719 (+ 362us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 14:32:25.155991 (+453272us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 14:32:25.156279 (+ 288us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60654: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":53}
W20260619 14:32:25.156680 7571 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60654: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260619 14:32:25.157181 7597 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:32:24.699056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:36025 (local address 127.0.0.1:60654)
0619 14:32:24.699417 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:24.699429 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:24.699526 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:24.699847 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:24.699849 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:24.699854 (+ 5us) client_negotiation.cc:189] Negotiated authn=JWT
0619 14:32:24.699954 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:24.699958 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:24.701065 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:24.701097 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:24.701734 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:24.701744 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:24.702394 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:24.702426 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:24.702501 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:24.702526 (+ 25us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 14:32:25.156289 (+453763us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 14:32:25.156558 (+ 269us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.2.62:36025: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":137,"threads_started":1}
I20260619 14:32:25.158195 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 7501
2026-06-19T14:32:25Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (740 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (128 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (127 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:32:25 dist-test-slave-70rm krb5kdc[7609](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:32:25 dist-test-slave-70rm krb5kdc[7609](info): set up 2 sockets
Jun 19 14:32:25 dist-test-slave-70rm krb5kdc[7609](info): commencing operation
krb5kdc: starting...
W20260619 14:32:26.466781 2056 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Jun 19 14:32:26 dist-test-slave-70rm krb5kdc[7609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:32:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:32:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.2.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab.
Entry for principal kudu/127.2.2.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab.
I20260619 14:32:26.591333 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:38127
--webserver_interface=127.2.2.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:35417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.2.62:38127
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:32:26.703253 7622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:26.703608 7622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:26.703699 7622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:26.707919 7622 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:32:26.708024 7622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:26.708063 7622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:26.708119 7622 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:32:26.708158 7622 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:32:26.713726 7622 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:35417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.2.62:38127
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:38127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.2.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:32:26.714931 7622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:26.716027 7622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:32:26.724848 7630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:26.724848 7628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:26.724874 7627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:26.725095 7622 server_base.cc:1034] running on GCE node
I20260619 14:32:26.725732 7622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:26.727180 7622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:26.728387 7622 hybrid_clock.cc:648] HybridClock initialized: now 1781879546728373 us; error 31 us; skew 500 ppm
Jun 19 14:32:26 dist-test-slave-70rm krb5kdc[7609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879546, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:32:26.731381 7622 init.cc:377] Logged in from keytab as kudu/127.2.2.62@KRBTEST.COM (short username kudu)
I20260619 14:32:26.732383 7622 webserver.cc:460] Webserver started at http://127.2.2.62:35695/ using document root <none> and password file <none>
I20260619 14:32:26.732980 7622 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:26.733071 7622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:26.733317 7622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:32:26.735508 7622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/master-0/data/instance:
uuid: "47c9dffeab4c4d94b45bb453ea629a98"
format_stamp: "Formatted at 2026-06-19 14:32:26 on dist-test-slave-70rm"
server_key: "af14b29a9ed35c74206f71622fefafdc"
server_key_iv: "0805d7f82cd301237e29e57e7bc35f5a"
server_key_version: "encryptionkey@0"
I20260619 14:32:26.736102 7622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/master-0/wal/instance:
uuid: "47c9dffeab4c4d94b45bb453ea629a98"
format_stamp: "Formatted at 2026-06-19 14:32:26 on dist-test-slave-70rm"
server_key: "af14b29a9ed35c74206f71622fefafdc"
server_key_iv: "0805d7f82cd301237e29e57e7bc35f5a"
server_key_version: "encryptionkey@0"
I20260619 14:32:26.740005 7622 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:32:26.742496 7637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:26.743759 7622 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:32:26.743903 7622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/master-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/master-0/wal
uuid: "47c9dffeab4c4d94b45bb453ea629a98"
format_stamp: "Formatted at 2026-06-19 14:32:26 on dist-test-slave-70rm"
server_key: "af14b29a9ed35c74206f71622fefafdc"
server_key_iv: "0805d7f82cd301237e29e57e7bc35f5a"
server_key_version: "encryptionkey@0"
I20260619 14:32:26.744019 7622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:26.757017 7622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:26.760210 7622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:32:26.760437 7622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:26.768491 7622 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.62:38127
I20260619 14:32:26.768517 7689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.62:38127 every 8 connection(s)
I20260619 14:32:26.769531 7622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
I20260619 14:32:26.772531 7690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:26.777976 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 7622
I20260619 14:32:26.778172 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/master-0/wal/instance
I20260619 14:32:26.778457 2056 external_mini_cluster.cc:1428] Setting key 853e98b0b4f9765e0a455b4805c585f6
I20260619 14:32:26.778649 7690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47c9dffeab4c4d94b45bb453ea629a98: Bootstrap starting.
I20260619 14:32:26.781337 7690 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47c9dffeab4c4d94b45bb453ea629a98: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:26.782182 7690 log.cc:824] T 00000000000000000000000000000000 P 47c9dffeab4c4d94b45bb453ea629a98: Log is configured to *not* fsync() on all Append() calls
I20260619 14:32:26.784544 7690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47c9dffeab4c4d94b45bb453ea629a98: No bootstrap required, opened a new log
Jun 19 14:32:26 dist-test-slave-70rm krb5kdc[7609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:32:26.789167 7690 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47c9dffeab4c4d94b45bb453ea629a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47c9dffeab4c4d94b45bb453ea629a98" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 38127 } }
I20260619 14:32:26.789505 7690 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47c9dffeab4c4d94b45bb453ea629a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:26.789582 7690 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47c9dffeab4c4d94b45bb453ea629a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47c9dffeab4c4d94b45bb453ea629a98, State: Initialized, Role: FOLLOWER
I20260619 14:32:26.790167 7690 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47c9dffeab4c4d94b45bb453ea629a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47c9dffeab4c4d94b45bb453ea629a98" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 38127 } }
I20260619 14:32:26.790347 7690 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47c9dffeab4c4d94b45bb453ea629a98 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:32:26.790422 7690 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47c9dffeab4c4d94b45bb453ea629a98 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:32:26.790524 7690 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47c9dffeab4c4d94b45bb453ea629a98 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:26.791749 7690 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47c9dffeab4c4d94b45bb453ea629a98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47c9dffeab4c4d94b45bb453ea629a98" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 38127 } }
I20260619 14:32:26.792148 7690 leader_election.cc:302] T 00000000000000000000000000000000 P 47c9dffeab4c4d94b45bb453ea629a98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47c9dffeab4c4d94b45bb453ea629a98; no voters:
I20260619 14:32:26.792516 7690 leader_election.cc:288] T 00000000000000000000000000000000 P 47c9dffeab4c4d94b45bb453ea629a98 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:32:26.792639 7695 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47c9dffeab4c4d94b45bb453ea629a98 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:32:26.792876 7695 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47c9dffeab4c4d94b45bb453ea629a98 [term 1 LEADER]: Becoming Leader. State: Replica: 47c9dffeab4c4d94b45bb453ea629a98, State: Running, Role: LEADER
I20260619 14:32:26.793251 7695 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47c9dffeab4c4d94b45bb453ea629a98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47c9dffeab4c4d94b45bb453ea629a98" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 38127 } }
I20260619 14:32:26.793910 7690 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47c9dffeab4c4d94b45bb453ea629a98 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:32:26.794706 7693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:26.779915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49930 (local address 127.2.2.62:38127)
0619 14:32:26.780408 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:26.780424 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:26.780461 (+ 37us) server_negotiation.cc:407] Connection header received
0619 14:32:26.781126 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:26.781145 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:26.781578 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:26.781935 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:26.782855 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:26.784282 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:26.785197 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:26.785889 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:26.786069 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:26.788867 (+ 2798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:26.788874 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:26.788892 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:26.788927 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:26.791141 (+ 2214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:26.791604 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:26.791613 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:26.791622 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:26.791774 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:26.792092 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:26.792095 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:26.792097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:26.792646 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:26.792860 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:26.793324 (+ 464us) server_negotiation.cc:299] Negotiation successful
0619 14:32:26.793640 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":144,"threads_started":1}
I20260619 14:32:26.795325 7696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47c9dffeab4c4d94b45bb453ea629a98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47c9dffeab4c4d94b45bb453ea629a98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47c9dffeab4c4d94b45bb453ea629a98" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 38127 } } }
I20260619 14:32:26.795439 7696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47c9dffeab4c4d94b45bb453ea629a98 [sys.catalog]: This master's current role is: LEADER
I20260619 14:32:26.795658 7697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47c9dffeab4c4d94b45bb453ea629a98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47c9dffeab4c4d94b45bb453ea629a98. Latest consensus state: current_term: 1 leader_uuid: "47c9dffeab4c4d94b45bb453ea629a98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47c9dffeab4c4d94b45bb453ea629a98" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 38127 } } }
I20260619 14:32:26.795745 7697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47c9dffeab4c4d94b45bb453ea629a98 [sys.catalog]: This master's current role is: LEADER
W20260619 14:32:26.799438 7710 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 47c9dffeab4c4d94b45bb453ea629a98: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:32:26.799546 7710 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:32:26.799618 7704 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:32:26.800350 7704 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:32:26.805649 7704 catalog_manager.cc:1347] Generated new cluster ID: 06071263e5df402d8ad02efd4fe5cd88
I20260619 14:32:26.805737 7704 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:32:26.821153 7704 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:32:26.822296 7704 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:32:26.830015 7704 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47c9dffeab4c4d94b45bb453ea629a98: Generated new TSK 0
I20260619 14:32:26.830878 7704 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.2.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab.
Entry for principal kudu/127.2.2.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab.
I20260619 14:32:26.871515 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.1:0
--local_ip_for_outbound_sockets=127.2.2.1
--webserver_interface=127.2.2.1
--webserver_port=0
--tserver_master_addrs=127.2.2.62:38127
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab
--principal=kudu/127.2.2.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:35417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:32:26.988078 7716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:26.988371 7716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:26.988415 7716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:26.992527 7716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:26.992599 7716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:26.992659 7716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.1
I20260619 14:32:26.998466 7716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:35417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab
--principal=kudu/127.2.2.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.2.1
--webserver_port=0
--tserver_master_addrs=127.2.2.62:38127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.1
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:32:26.999545 7716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:27.000576 7716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:32:27.010363 7721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:27.010479 7724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:27.010448 7722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:27.010586 7716 server_base.cc:1034] running on GCE node
I20260619 14:32:27.011123 7716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:27.011808 7716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:27.012993 7716 hybrid_clock.cc:648] HybridClock initialized: now 1781879547012972 us; error 33 us; skew 500 ppm
Jun 19 14:32:27 dist-test-slave-70rm krb5kdc[7609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879547, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:32:27.015854 7716 init.cc:377] Logged in from keytab as kudu/127.2.2.1@KRBTEST.COM (short username kudu)
I20260619 14:32:27.016827 7716 webserver.cc:460] Webserver started at http://127.2.2.1:46681/ using document root <none> and password file <none>
I20260619 14:32:27.017493 7716 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:27.017576 7716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:27.017820 7716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:32:27.020020 7716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/ts-0/data/instance:
uuid: "b297796ebb40401fb74ad7296b0ea742"
format_stamp: "Formatted at 2026-06-19 14:32:27 on dist-test-slave-70rm"
server_key: "69e6378c1848ad7cf2defeb89d61c4e3"
server_key_iv: "b746c50dc9b9f99fbb32abe19faf7564"
server_key_version: "encryptionkey@0"
I20260619 14:32:27.020619 7716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/ts-0/wal/instance:
uuid: "b297796ebb40401fb74ad7296b0ea742"
format_stamp: "Formatted at 2026-06-19 14:32:27 on dist-test-slave-70rm"
server_key: "69e6378c1848ad7cf2defeb89d61c4e3"
server_key_iv: "b746c50dc9b9f99fbb32abe19faf7564"
server_key_version: "encryptionkey@0"
I20260619 14:32:27.024698 7716 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:32:27.027413 7731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:27.028777 7716 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 14:32:27.028934 7716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/ts-0/wal
uuid: "b297796ebb40401fb74ad7296b0ea742"
format_stamp: "Formatted at 2026-06-19 14:32:27 on dist-test-slave-70rm"
server_key: "69e6378c1848ad7cf2defeb89d61c4e3"
server_key_iv: "b746c50dc9b9f99fbb32abe19faf7564"
server_key_version: "encryptionkey@0"
I20260619 14:32:27.029063 7716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:27.048400 7716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:27.051611 7716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:32:27.051827 7716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:27.052451 7716 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:32:27.053298 7716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:32:27.053370 7716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:27.053444 7716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:32:27.053496 7716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:27.063022 7716 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.1:35991
I20260619 14:32:27.064144 7716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
I20260619 14:32:27.064248 7844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.1:35991 every 8 connection(s)
I20260619 14:32:27.072593 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 7716
I20260619 14:32:27.072750 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/ts-0/wal/instance
I20260619 14:32:27.073148 2056 external_mini_cluster.cc:1428] Setting key 43cc1da632628756d8f4d492b74beec9
Jun 19 14:32:27 dist-test-slave-70rm krb5kdc[7609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879547, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.1@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:32:27.080511 7693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:27.066459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:49527 (local address 127.2.2.62:38127)
0619 14:32:27.066605 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:27.066610 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:27.067762 (+ 1152us) server_negotiation.cc:407] Connection header received
0619 14:32:27.069236 (+ 1474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:27.069240 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:27.069299 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:27.069382 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:27.070694 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:27.071251 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:27.072460 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:27.072756 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:27.072889 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:27.076459 (+ 3570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:27.076462 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:27.076464 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:27.076488 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:27.078056 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:27.078739 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:27.078742 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:27.078743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:27.078820 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:27.079276 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:27.079279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:27.079280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:27.079462 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:27.079536 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:27.080271 (+ 735us) server_negotiation.cc:299] Negotiation successful
0619 14:32:27.080384 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 14:32:27.081856 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:27.067075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:38127 (local address 127.2.2.1:49527)
0619 14:32:27.067591 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:27.067637 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:27.069002 (+ 1365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:27.069572 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:27.069579 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:27.069954 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:27.070539 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:27.070553 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:27.071409 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:27.071412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:27.072320 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:27.072329 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:27.072903 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:27.072971 (+ 68us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:27.073244 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:27.074062 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:27.074083 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:27.076278 (+ 2195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:27.078206 (+ 1928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:27.078216 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:27.078236 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:27.078619 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:27.078914 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:27.078921 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:27.078926 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:27.079168 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:27.079569 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:27.079578 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:27.079968 (+ 390us) client_negotiation.cc:769] Sending connection context
0619 14:32:27.080220 (+ 252us) client_negotiation.cc:240] Negotiation successful
0619 14:32:27.080648 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":120,"threads_started":1}
I20260619 14:32:27.083217 7845 heartbeater.cc:347] Connected to a master server at 127.2.2.62:38127
WARNING: no policy specified for kudu/127.2.2.2@KRBTEST.COM; defaulting to no policy
I20260619 14:32:27.083459 7845 heartbeater.cc:464] Registering TS with master...
I20260619 14:32:27.083984 7845 heartbeater.cc:511] Master 127.2.2.62:38127 requested a full tablet report, sending...
I20260619 14:32:27.085506 7654 ts_manager.cc:194] Registered new tserver with Master: b297796ebb40401fb74ad7296b0ea742 (127.2.2.1:35991)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.2@KRBTEST.COM" created.
I20260619 14:32:27.087077 7654 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.2.1@KRBTEST.COM'} at 127.2.2.1:49527
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab.
Entry for principal kudu/127.2.2.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab.
I20260619 14:32:27.101552 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.2:0
--local_ip_for_outbound_sockets=127.2.2.2
--webserver_interface=127.2.2.2
--webserver_port=0
--tserver_master_addrs=127.2.2.62:38127
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab
--principal=kudu/127.2.2.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:35417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:32:27.214969 7850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:27.215266 7850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:27.215310 7850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:27.219485 7850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:27.219558 7850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:27.219650 7850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.2
I20260619 14:32:27.225122 7850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:35417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab
--principal=kudu/127.2.2.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.2.2
--webserver_port=0
--tserver_master_addrs=127.2.2.62:38127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.2
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:32:27.226310 7850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:27.227367 7850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:32:27.237483 7855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:27.237593 7850 server_base.cc:1034] running on GCE node
W20260619 14:32:27.237483 7858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:27.237486 7856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:27.238101 7850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:27.238834 7850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:27.239992 7850 hybrid_clock.cc:648] HybridClock initialized: now 1781879547239941 us; error 65 us; skew 500 ppm
Jun 19 14:32:27 dist-test-slave-70rm krb5kdc[7609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879547, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:32:27.243086 7850 init.cc:377] Logged in from keytab as kudu/127.2.2.2@KRBTEST.COM (short username kudu)
I20260619 14:32:27.244057 7850 webserver.cc:460] Webserver started at http://127.2.2.2:36117/ using document root <none> and password file <none>
I20260619 14:32:27.244649 7850 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:27.244719 7850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:27.244982 7850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:32:27.247368 7850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/ts-1/data/instance:
uuid: "641e2ce2af2f4afb870bb98170ce0932"
format_stamp: "Formatted at 2026-06-19 14:32:27 on dist-test-slave-70rm"
server_key: "887abbb16a913b0ef53aaf845cd26995"
server_key_iv: "53d37adfa1711108f6c9258e41af96c1"
server_key_version: "encryptionkey@0"
I20260619 14:32:27.248018 7850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/ts-1/wal/instance:
uuid: "641e2ce2af2f4afb870bb98170ce0932"
format_stamp: "Formatted at 2026-06-19 14:32:27 on dist-test-slave-70rm"
server_key: "887abbb16a913b0ef53aaf845cd26995"
server_key_iv: "53d37adfa1711108f6c9258e41af96c1"
server_key_version: "encryptionkey@0"
I20260619 14:32:27.252178 7850 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:32:27.254875 7865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:27.256186 7850 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:32:27.256346 7850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/ts-1/wal
uuid: "641e2ce2af2f4afb870bb98170ce0932"
format_stamp: "Formatted at 2026-06-19 14:32:27 on dist-test-slave-70rm"
server_key: "887abbb16a913b0ef53aaf845cd26995"
server_key_iv: "53d37adfa1711108f6c9258e41af96c1"
server_key_version: "encryptionkey@0"
I20260619 14:32:27.256472 7850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:27.272722 7850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:27.275969 7850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:32:27.276196 7850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:27.276850 7850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:32:27.277768 7850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:32:27.277817 7850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:27.277885 7850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:32:27.277918 7850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:27.287292 7850 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.2:46455
I20260619 14:32:27.287310 7978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.2:46455 every 8 connection(s)
I20260619 14:32:27.288332 7850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
I20260619 14:32:27.298161 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 7850
I20260619 14:32:27.298287 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/ts-1/wal/instance
I20260619 14:32:27.298579 2056 external_mini_cluster.cc:1428] Setting key a250919b40bb1124df1085ae76f843bf
Jun 19 14:32:27 dist-test-slave-70rm krb5kdc[7609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879547, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.2@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:32:27.303462 7693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:27.290208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:49413 (local address 127.2.2.62:38127)
0619 14:32:27.290348 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:27.290352 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:27.291361 (+ 1009us) server_negotiation.cc:407] Connection header received
0619 14:32:27.292709 (+ 1348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:27.292720 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:27.292770 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:27.292849 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:27.294117 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:27.294860 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:27.296257 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:27.296479 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:27.296569 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:27.299366 (+ 2797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:27.299370 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:27.299372 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:27.299396 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:27.301159 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:27.301781 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:27.301785 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:27.301787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:27.301884 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:27.302317 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:27.302321 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:27.302323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:27.302523 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:27.302630 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:27.303161 (+ 531us) server_negotiation.cc:299] Negotiation successful
0619 14:32:27.303290 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 14:32:27.304420 7981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:27.290725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:38127 (local address 127.2.2.2:49413)
0619 14:32:27.291193 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:27.291240 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:27.292500 (+ 1260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:27.293024 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:27.293031 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:27.293394 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:27.293950 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:27.293964 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:27.295027 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:27.295030 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:27.296126 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:27.296136 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:27.296595 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:27.296670 (+ 75us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:27.296870 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:27.297436 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:27.297454 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:27.299184 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:27.301314 (+ 2130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:27.301320 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:27.301334 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:27.301638 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:27.301981 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:27.301986 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:27.301988 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:27.302210 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:27.302630 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:27.302638 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:27.302908 (+ 270us) client_negotiation.cc:769] Sending connection context
0619 14:32:27.303111 (+ 203us) client_negotiation.cc:240] Negotiation successful
0619 14:32:27.303462 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":116,"threads_started":1}
I20260619 14:32:27.305907 7979 heartbeater.cc:347] Connected to a master server at 127.2.2.62:38127
I20260619 14:32:27.306268 7979 heartbeater.cc:464] Registering TS with master...
I20260619 14:32:27.306797 7979 heartbeater.cc:511] Master 127.2.2.62:38127 requested a full tablet report, sending...
I20260619 14:32:27.307979 7654 ts_manager.cc:194] Registered new tserver with Master: 641e2ce2af2f4afb870bb98170ce0932 (127.2.2.2:46455)
WARNING: no policy specified for kudu/127.2.2.3@KRBTEST.COM; defaulting to no policy
I20260619 14:32:27.308625 7654 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.2.2@KRBTEST.COM'} at 127.2.2.2:49413
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab.
Entry for principal kudu/127.2.2.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab.
I20260619 14:32:27.325800 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.3:0
--local_ip_for_outbound_sockets=127.2.2.3
--webserver_interface=127.2.2.3
--webserver_port=0
--tserver_master_addrs=127.2.2.62:38127
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab
--principal=kudu/127.2.2.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:35417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:32:27.439085 7984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:27.439357 7984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:27.439401 7984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:27.443554 7984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:27.443616 7984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:27.443675 7984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.3
I20260619 14:32:27.449172 7984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:35417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab
--principal=kudu/127.2.2.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.2.3
--webserver_port=0
--tserver_master_addrs=127.2.2.62:38127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.3
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:32:27.450249 7984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:27.451318 7984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:32:27.460719 7990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:27.460722 7989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:27.460716 7992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:27.461148 7984 server_base.cc:1034] running on GCE node
I20260619 14:32:27.461565 7984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:27.462263 7984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:27.463441 7984 hybrid_clock.cc:648] HybridClock initialized: now 1781879547463420 us; error 38 us; skew 500 ppm
Jun 19 14:32:27 dist-test-slave-70rm krb5kdc[7609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879547, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:32:27.466470 7984 init.cc:377] Logged in from keytab as kudu/127.2.2.3@KRBTEST.COM (short username kudu)
I20260619 14:32:27.467489 7984 webserver.cc:460] Webserver started at http://127.2.2.3:44625/ using document root <none> and password file <none>
I20260619 14:32:27.468143 7984 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:27.468243 7984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:27.468484 7984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:32:27.470683 7984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/ts-2/data/instance:
uuid: "052af00290dc4b34bf015de726766f72"
format_stamp: "Formatted at 2026-06-19 14:32:27 on dist-test-slave-70rm"
server_key: "66d821efab5cb2c42277b03708877e38"
server_key_iv: "784652a9c34c54478881c26ae6e2a2a0"
server_key_version: "encryptionkey@0"
I20260619 14:32:27.471282 7984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/ts-2/wal/instance:
uuid: "052af00290dc4b34bf015de726766f72"
format_stamp: "Formatted at 2026-06-19 14:32:27 on dist-test-slave-70rm"
server_key: "66d821efab5cb2c42277b03708877e38"
server_key_iv: "784652a9c34c54478881c26ae6e2a2a0"
server_key_version: "encryptionkey@0"
I20260619 14:32:27.475258 7984 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:32:27.477896 7999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:27.479228 7984 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:32:27.479377 7984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/ts-2/wal
uuid: "052af00290dc4b34bf015de726766f72"
format_stamp: "Formatted at 2026-06-19 14:32:27 on dist-test-slave-70rm"
server_key: "66d821efab5cb2c42277b03708877e38"
server_key_iv: "784652a9c34c54478881c26ae6e2a2a0"
server_key_version: "encryptionkey@0"
I20260619 14:32:27.479503 7984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:27.511749 7984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:27.515242 7984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:32:27.515455 7984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:27.516064 7984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:32:27.516921 7984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:32:27.516990 7984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:27.517062 7984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:32:27.517112 7984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:27.527159 7984 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.3:34647
I20260619 14:32:27.527179 8112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.3:34647 every 8 connection(s)
I20260619 14:32:27.528143 7984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
I20260619 14:32:27.532609 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 7984
I20260619 14:32:27.532757 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/minicluster-data/ts-2/wal/instance
I20260619 14:32:27.533031 2056 external_mini_cluster.cc:1428] Setting key 4cf20bc5817698ee085d9a1d22ad5412
Jun 19 14:32:27 dist-test-slave-70rm krb5kdc[7609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879547, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.3@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:32:27.542953 7693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:27.529997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.3:45173 (local address 127.2.2.62:38127)
0619 14:32:27.530228 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:27.530233 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:27.531154 (+ 921us) server_negotiation.cc:407] Connection header received
0619 14:32:27.532711 (+ 1557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:27.532715 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:27.532776 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:27.532858 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:27.534570 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:27.535169 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:27.536340 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:27.536576 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:27.536690 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:27.539404 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:27.539408 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:27.539417 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:27.539443 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:27.540708 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:27.541277 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:27.541281 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:27.541282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:27.541366 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:27.541720 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:27.541723 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:27.541725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:27.541907 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:27.542016 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:27.542663 (+ 647us) server_negotiation.cc:299] Negotiation successful
0619 14:32:27.542789 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 14:32:27.543794 8115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:27.530468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:38127 (local address 127.2.2.3:45173)
0619 14:32:27.530965 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:27.531011 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:27.532444 (+ 1433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:27.533116 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:27.533128 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:27.533674 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:27.534414 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:27.534443 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:27.535291 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:27.535294 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:27.536213 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:27.536221 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:27.536675 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:27.536733 (+ 58us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:27.536934 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:27.537590 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:27.537607 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:27.539218 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:27.540833 (+ 1615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:27.540840 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:27.540857 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:27.541167 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:27.541466 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:27.541469 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:27.541471 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:27.541617 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:27.542012 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:27.542020 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:27.542402 (+ 382us) client_negotiation.cc:769] Sending connection context
0619 14:32:27.542609 (+ 207us) client_negotiation.cc:240] Negotiation successful
0619 14:32:27.542952 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":114,"threads_started":1}
I20260619 14:32:27.544889 8113 heartbeater.cc:347] Connected to a master server at 127.2.2.62:38127
I20260619 14:32:27.545145 8113 heartbeater.cc:464] Registering TS with master...
I20260619 14:32:27.545733 8113 heartbeater.cc:511] Master 127.2.2.62:38127 requested a full tablet report, sending...
I20260619 14:32:27.546953 7653 ts_manager.cc:194] Registered new tserver with Master: 052af00290dc4b34bf015de726766f72 (127.2.2.3:34647)
I20260619 14:32:27.547575 7653 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.2.3@KRBTEST.COM'} at 127.2.2.3:45173
I20260619 14:32:27.559192 2056 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/krb5kdc/test-admin.keytab.
Jun 19 14:32:27 dist-test-slave-70rm krb5kdc[7609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:32:27.575129 2056 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260619 14:32:27.578186 8120 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:32:27.576609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37652 (local address 127.2.2.1:35991)
0619 14:32:27.576929 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:27.576945 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:27.576971 (+ 26us) server_negotiation.cc:407] Connection header received
0619 14:32:27.577031 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:27.577037 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:27.577238 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:27.577374 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:27.577952 (+ 578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37652: BlockingRecv error: recv got EOF from 127.0.0.1:37652 (error 108)
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":98,"threads_started":1}
W20260619 14:32:27.578176 2056 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/krb5kdc/krb5cc' of type FILE
Jun 19 14:32:27 dist-test-slave-70rm krb5kdc[7609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:32:27.579258 2056 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 14:32:27 dist-test-slave-70rm krb5kdc[7609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:32:27.580133 2056 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 14:32:27 dist-test-slave-70rm krb5kdc[7609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.2.1@KRBTEST.COM
I20260619 14:32:27.598196 8120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:27.582520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37656 (local address 127.2.2.1:35991)
0619 14:32:27.582667 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:27.582674 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:27.582833 (+ 159us) server_negotiation.cc:407] Connection header received
0619 14:32:27.582923 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:27.582927 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:27.582988 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:27.583083 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:27.583828 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:27.585088 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:27.586396 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:27.587094 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:27.587203 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:27.594727 (+ 7524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:27.594739 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:27.594754 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:27.594784 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:27.596484 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:27.596937 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:27.596943 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:27.596944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:27.597024 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:27.597274 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:27.597276 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:27.597277 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:27.597528 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:27.597700 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:27.597851 (+ 151us) server_negotiation.cc:299] Negotiation successful
0619 14:32:27.597979 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
W20260619 14:32:27.602212 8120 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:32:27.601122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37670 (local address 127.2.2.1:35991)
0619 14:32:27.601254 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:27.601259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:27.601372 (+ 113us) server_negotiation.cc:407] Connection header received
0619 14:32:27.601519 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:27.601521 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:27.601558 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:27.601642 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:27.602056 (+ 414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37670: BlockingRecv error: recv got EOF from 127.0.0.1:37670 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260619 14:32:27.602329 2056 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/krb5kdc/krb5cc' of type FILE
Jun 19 14:32:27 dist-test-slave-70rm krb5kdc[7609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:32:27.603410 2056 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 14:32:27 dist-test-slave-70rm krb5kdc[7609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:32:27.604310 2056 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 14:32:27 dist-test-slave-70rm krb5kdc[7609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.2.1@KRBTEST.COM
I20260619 14:32:27.613298 8120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:27.605116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37676 (local address 127.2.2.1:35991)
0619 14:32:27.605236 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:27.605240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:27.605363 (+ 123us) server_negotiation.cc:407] Connection header received
0619 14:32:27.605454 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:27.605457 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:27.605492 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:27.605557 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:27.606003 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:27.606611 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:27.607401 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:27.607659 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:27.607742 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:27.610309 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:27.610312 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:27.610314 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:27.610343 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:27.611773 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:27.612163 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:27.612165 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:27.612166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:27.612249 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:27.612506 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:27.612509 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:27.612511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:27.612704 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:27.612825 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:27.612976 (+ 151us) server_negotiation.cc:299] Negotiation successful
0619 14:32:27.613077 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260619 14:32:27.616079 8120 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:32:27.615073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37678 (local address 127.2.2.1:35991)
0619 14:32:27.615218 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:27.615224 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:27.615322 (+ 98us) server_negotiation.cc:407] Connection header received
0619 14:32:27.615406 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:27.615408 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:27.615442 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:27.615505 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:27.615964 (+ 459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37678: BlockingRecv error: recv got EOF from 127.0.0.1:37678 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260619 14:32:27.616348 2056 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879514993456-2056-0/krb5kdc/krb5cc)
Jun 19 14:32:27 dist-test-slave-70rm krb5kdc[7609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:32:27.617408 2056 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 14:32:27 dist-test-slave-70rm krb5kdc[7609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:32:27.618394 2056 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 14:32:27 dist-test-slave-70rm krb5kdc[7609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.2.1@KRBTEST.COM
I20260619 14:32:27.627169 8120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:27.619359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37684 (local address 127.2.2.1:35991)
0619 14:32:27.619480 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:27.619486 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:27.619594 (+ 108us) server_negotiation.cc:407] Connection header received
0619 14:32:27.619687 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:27.619689 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:27.619725 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:27.619789 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:27.620265 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:27.620758 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:27.621556 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:27.621782 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:27.621878 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:27.624110 (+ 2232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:27.624113 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:27.624115 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:27.624137 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:27.625621 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:27.626018 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:27.626021 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:27.626022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:27.626100 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:27.626405 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:27.626407 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:27.626409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:27.626575 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:27.626706 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:27.626860 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 14:32:27.626948 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 14:32:27.628192 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 7716
I20260619 14:32:27.633960 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 7850
I20260619 14:32:27.639353 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 7984
I20260619 14:32:27.644551 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 7622
2026-06-19T14:32:27Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2232 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:32:27 dist-test-slave-70rm krb5kdc[8133](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:32:27 dist-test-slave-70rm krb5kdc[8133](info): set up 2 sockets
Jun 19 14:32:27 dist-test-slave-70rm krb5kdc[8133](info): commencing operation
krb5kdc: starting...
W20260619 14:32:28.698549 2056 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Jun 19 14:32:28 dist-test-slave-70rm krb5kdc[8133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:32:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:32:28Z Disabled control of system clock
WARNING: no policy specified for oryx/127.2.2.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.2.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.2.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/krb5kdc/oryx_127.2.2.62.keytab.
Entry for principal oryx/127.2.2.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/krb5kdc/oryx_127.2.2.62.keytab.
I20260619 14:32:28.825290 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:39059
--webserver_interface=127.2.2.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/krb5kdc/oryx_127.2.2.62.keytab
--principal=oryx/127.2.2.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:45229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.2.62:39059
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:32:28.939445 8146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:28.939801 8146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:28.939889 8146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:28.943998 8146 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:32:28.944119 8146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:28.944165 8146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:28.944208 8146 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:32:28.944248 8146 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:32:28.949661 8146 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:45229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.2.62:39059
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/krb5kdc/oryx_127.2.2.62.keytab
--principal=oryx/127.2.2.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:39059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.2.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:32:28.950898 8146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:28.952011 8146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:32:28.961040 8154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:28.961015 8152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:28.961089 8151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:28.961189 8146 server_base.cc:1034] running on GCE node
I20260619 14:32:28.961838 8146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:28.963080 8146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:28.964277 8146 hybrid_clock.cc:648] HybridClock initialized: now 1781879548964261 us; error 36 us; skew 500 ppm
Jun 19 14:32:28 dist-test-slave-70rm krb5kdc[8133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879548, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.2.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:32:28.967070 8146 init.cc:377] Logged in from keytab as oryx/127.2.2.62@KRBTEST.COM (short username oryx)
I20260619 14:32:28.968019 8146 webserver.cc:460] Webserver started at http://127.2.2.62:39465/ using document root <none> and password file <none>
I20260619 14:32:28.968586 8146 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:28.968676 8146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:28.968909 8146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:32:28.971133 8146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-0/data/instance:
uuid: "cf27644a9d194a898a1d93d6821e8b38"
format_stamp: "Formatted at 2026-06-19 14:32:28 on dist-test-slave-70rm"
server_key: "dc97b96580d97ea13877e0c09380c621"
server_key_iv: "249b8655209c41d17b273a97527bbbf0"
server_key_version: "encryptionkey@0"
I20260619 14:32:28.971714 8146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-0/wal/instance:
uuid: "cf27644a9d194a898a1d93d6821e8b38"
format_stamp: "Formatted at 2026-06-19 14:32:28 on dist-test-slave-70rm"
server_key: "dc97b96580d97ea13877e0c09380c621"
server_key_iv: "249b8655209c41d17b273a97527bbbf0"
server_key_version: "encryptionkey@0"
I20260619 14:32:28.975481 8146 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:32:28.977846 8161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:28.979207 8146 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:32:28.979336 8146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-0/wal
uuid: "cf27644a9d194a898a1d93d6821e8b38"
format_stamp: "Formatted at 2026-06-19 14:32:28 on dist-test-slave-70rm"
server_key: "dc97b96580d97ea13877e0c09380c621"
server_key_iv: "249b8655209c41d17b273a97527bbbf0"
server_key_version: "encryptionkey@0"
I20260619 14:32:28.979457 8146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:28.990785 8146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:28.993846 8146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:32:28.994052 8146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:29.000785 8146 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.62:39059
I20260619 14:32:29.001097 8213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.62:39059 every 8 connection(s)
I20260619 14:32:29.001858 8146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
I20260619 14:32:29.004786 8214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:29.010540 8214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf27644a9d194a898a1d93d6821e8b38: Bootstrap starting.
I20260619 14:32:29.011911 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 8146
I20260619 14:32:29.012022 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-0/wal/instance
I20260619 14:32:29.012306 2056 external_mini_cluster.cc:1428] Setting key f6bd934faaf3548b125dcaeab9aaec0b
I20260619 14:32:29.013257 8214 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf27644a9d194a898a1d93d6821e8b38: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:29.014111 8214 log.cc:824] T 00000000000000000000000000000000 P cf27644a9d194a898a1d93d6821e8b38: Log is configured to *not* fsync() on all Append() calls
I20260619 14:32:29.016098 8214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf27644a9d194a898a1d93d6821e8b38: No bootstrap required, opened a new log
I20260619 14:32:29.020525 8214 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf27644a9d194a898a1d93d6821e8b38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf27644a9d194a898a1d93d6821e8b38" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 39059 } }
I20260619 14:32:29.020759 8214 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf27644a9d194a898a1d93d6821e8b38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:29.020794 8214 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf27644a9d194a898a1d93d6821e8b38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf27644a9d194a898a1d93d6821e8b38, State: Initialized, Role: FOLLOWER
I20260619 14:32:29.021263 8214 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf27644a9d194a898a1d93d6821e8b38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf27644a9d194a898a1d93d6821e8b38" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 39059 } }
I20260619 14:32:29.021421 8214 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf27644a9d194a898a1d93d6821e8b38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:32:29.021510 8214 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf27644a9d194a898a1d93d6821e8b38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:32:29.021636 8214 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf27644a9d194a898a1d93d6821e8b38 [term 0 FOLLOWER]: Advancing to term 1
Jun 19 14:32:29 dist-test-slave-70rm krb5kdc[8133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM
I20260619 14:32:29.022836 8214 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf27644a9d194a898a1d93d6821e8b38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf27644a9d194a898a1d93d6821e8b38" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 39059 } }
I20260619 14:32:29.023276 8214 leader_election.cc:302] T 00000000000000000000000000000000 P cf27644a9d194a898a1d93d6821e8b38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf27644a9d194a898a1d93d6821e8b38; no voters:
I20260619 14:32:29.023661 8214 leader_election.cc:288] T 00000000000000000000000000000000 P cf27644a9d194a898a1d93d6821e8b38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:32:29.023950 8219 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf27644a9d194a898a1d93d6821e8b38 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:32:29.024255 8219 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf27644a9d194a898a1d93d6821e8b38 [term 1 LEADER]: Becoming Leader. State: Replica: cf27644a9d194a898a1d93d6821e8b38, State: Running, Role: LEADER
I20260619 14:32:29.024612 8219 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf27644a9d194a898a1d93d6821e8b38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf27644a9d194a898a1d93d6821e8b38" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 39059 } }
I20260619 14:32:29.025106 8214 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf27644a9d194a898a1d93d6821e8b38 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:32:29.026423 8220 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf27644a9d194a898a1d93d6821e8b38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf27644a9d194a898a1d93d6821e8b38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf27644a9d194a898a1d93d6821e8b38" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 39059 } } }
I20260619 14:32:29.026551 8220 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf27644a9d194a898a1d93d6821e8b38 [sys.catalog]: This master's current role is: LEADER
I20260619 14:32:29.026952 8221 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf27644a9d194a898a1d93d6821e8b38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf27644a9d194a898a1d93d6821e8b38. Latest consensus state: current_term: 1 leader_uuid: "cf27644a9d194a898a1d93d6821e8b38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf27644a9d194a898a1d93d6821e8b38" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 39059 } } }
I20260619 14:32:29.027099 8221 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf27644a9d194a898a1d93d6821e8b38 [sys.catalog]: This master's current role is: LEADER
I20260619 14:32:29.029178 8217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:29.013396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37734 (local address 127.2.2.62:39059)
0619 14:32:29.013886 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:29.013901 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:29.013937 (+ 36us) server_negotiation.cc:407] Connection header received
0619 14:32:29.014732 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:29.014752 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:29.015091 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:29.015443 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:29.016281 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:29.017779 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:29.018665 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:29.019512 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:29.019683 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:29.022485 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:29.022495 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:29.022517 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:29.022553 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:29.024825 (+ 2272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:29.025370 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:29.025376 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:29.025382 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:29.025539 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:29.025890 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:29.025893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:29.025895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:29.026523 (+ 628us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:29.027927 (+ 1404us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:29.028054 (+ 127us) server_negotiation.cc:299] Negotiation successful
0619 14:32:29.028303 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":164,"threads_started":1}
W20260619 14:32:29.030409 8234 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cf27644a9d194a898a1d93d6821e8b38: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:32:29.030511 8234 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:32:29.030637 8228 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:32:29.031601 8228 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:32:29.037381 8228 catalog_manager.cc:1347] Generated new cluster ID: 96382bd00e6e4b0792b1bfa035e796dc
I20260619 14:32:29.037468 8228 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:32:29.046401 8228 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:32:29.047658 8228 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:32:29.059724 8228 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf27644a9d194a898a1d93d6821e8b38: Generated new TSK 0
I20260619 14:32:29.060617 8228 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.2.2.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.2.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.2.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/krb5kdc/oryx_127.2.2.1.keytab.
Entry for principal oryx/127.2.2.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/krb5kdc/oryx_127.2.2.1.keytab.
I20260619 14:32:29.093230 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.1:0
--local_ip_for_outbound_sockets=127.2.2.1
--webserver_interface=127.2.2.1
--webserver_port=0
--tserver_master_addrs=127.2.2.62:39059
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/krb5kdc/oryx_127.2.2.1.keytab
--principal=oryx/127.2.2.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:45229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:32:29.210897 8240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:29.211189 8240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:29.211236 8240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:29.215401 8240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:29.215477 8240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:29.215540 8240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.1
I20260619 14:32:29.221130 8240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:45229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/krb5kdc/oryx_127.2.2.1.keytab
--principal=oryx/127.2.2.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.2.1
--webserver_port=0
--tserver_master_addrs=127.2.2.62:39059
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.1
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:32:29.222342 8240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:29.223400 8240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:32:29.233764 8246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:29.233708 8245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:29.233708 8248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:29.234032 8240 server_base.cc:1034] running on GCE node
I20260619 14:32:29.234525 8240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:29.235217 8240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:29.236405 8240 hybrid_clock.cc:648] HybridClock initialized: now 1781879549236387 us; error 32 us; skew 500 ppm
Jun 19 14:32:29 dist-test-slave-70rm krb5kdc[8133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879549, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.2.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:32:29.239284 8240 init.cc:377] Logged in from keytab as oryx/127.2.2.1@KRBTEST.COM (short username oryx)
I20260619 14:32:29.240165 8240 webserver.cc:460] Webserver started at http://127.2.2.1:45677/ using document root <none> and password file <none>
I20260619 14:32:29.240748 8240 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:29.240836 8240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:29.241067 8240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:32:29.243227 8240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-0/data/instance:
uuid: "c81bd9d0e5f44e0fa56ac4dcc2401a27"
format_stamp: "Formatted at 2026-06-19 14:32:29 on dist-test-slave-70rm"
server_key: "60f5eaed1e95d8c80339ac3dfbedf157"
server_key_iv: "d4810e94317241558a02157c1a570b77"
server_key_version: "encryptionkey@0"
I20260619 14:32:29.243806 8240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-0/wal/instance:
uuid: "c81bd9d0e5f44e0fa56ac4dcc2401a27"
format_stamp: "Formatted at 2026-06-19 14:32:29 on dist-test-slave-70rm"
server_key: "60f5eaed1e95d8c80339ac3dfbedf157"
server_key_iv: "d4810e94317241558a02157c1a570b77"
server_key_version: "encryptionkey@0"
I20260619 14:32:29.247782 8240 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:32:29.250370 8255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:29.251726 8240 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:32:29.251885 8240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-0/wal
uuid: "c81bd9d0e5f44e0fa56ac4dcc2401a27"
format_stamp: "Formatted at 2026-06-19 14:32:29 on dist-test-slave-70rm"
server_key: "60f5eaed1e95d8c80339ac3dfbedf157"
server_key_iv: "d4810e94317241558a02157c1a570b77"
server_key_version: "encryptionkey@0"
I20260619 14:32:29.252020 8240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:29.264856 8240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:29.268074 8240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:32:29.268306 8240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:29.268931 8240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:32:29.269876 8240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:32:29.269945 8240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:29.270023 8240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:32:29.270066 8240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:29.280259 8240 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.1:45703
I20260619 14:32:29.280293 8368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.1:45703 every 8 connection(s)
I20260619 14:32:29.281257 8240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
I20260619 14:32:29.290021 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 8240
I20260619 14:32:29.290205 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-0/wal/instance
I20260619 14:32:29.290480 2056 external_mini_cluster.cc:1428] Setting key 4adfc0c734bff2e229138617d1c7db7d
Jun 19 14:32:29 dist-test-slave-70rm krb5kdc[8133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879549, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.2.1@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM
I20260619 14:32:29.296012 8217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:29.283223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:43479 (local address 127.2.2.62:39059)
0619 14:32:29.283374 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:29.283378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:29.284455 (+ 1077us) server_negotiation.cc:407] Connection header received
0619 14:32:29.285819 (+ 1364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:29.285822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:29.285874 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:29.285957 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:29.287257 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:29.287803 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:29.288937 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:29.289172 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:29.289258 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:29.292178 (+ 2920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:29.292181 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:29.292183 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:29.292207 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:29.293615 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:29.294174 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:29.294177 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:29.294179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:29.294268 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:29.294633 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:29.294637 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:29.294642 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:29.294898 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:29.295005 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:29.295597 (+ 592us) server_negotiation.cc:299] Negotiation successful
0619 14:32:29.295748 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 14:32:29.296774 8371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:29.283712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:39059 (local address 127.2.2.1:43479)
0619 14:32:29.284247 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:29.284292 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:29.285582 (+ 1290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:29.286164 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:29.286173 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:29.286565 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:29.287095 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:29.287109 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:29.287921 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:29.287924 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:29.288825 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:29.288833 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:29.289265 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:29.289317 (+ 52us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:29.289462 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:29.290170 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:29.290195 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:29.292003 (+ 1808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:29.293740 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:29.293746 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:29.293764 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:29.294037 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:29.294373 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:29.294376 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:29.294379 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:29.294539 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:29.295024 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:29.295031 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:29.295332 (+ 301us) client_negotiation.cc:769] Sending connection context
0619 14:32:29.295518 (+ 186us) client_negotiation.cc:240] Negotiation successful
0619 14:32:29.295822 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":129,"threads_started":1}
I20260619 14:32:29.298370 8369 heartbeater.cc:347] Connected to a master server at 127.2.2.62:39059
I20260619 14:32:29.298630 8369 heartbeater.cc:464] Registering TS with master...
I20260619 14:32:29.299171 8369 heartbeater.cc:511] Master 127.2.2.62:39059 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.2.2.2@KRBTEST.COM; defaulting to no policy
I20260619 14:32:29.301019 8174 ts_manager.cc:194] Registered new tserver with Master: c81bd9d0e5f44e0fa56ac4dcc2401a27 (127.2.2.1:45703)
I20260619 14:32:29.302264 8174 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.2.1@KRBTEST.COM'} at 127.2.2.1:43479
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.2.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.2.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/krb5kdc/oryx_127.2.2.2.keytab.
Entry for principal oryx/127.2.2.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/krb5kdc/oryx_127.2.2.2.keytab.
I20260619 14:32:29.318708 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.2:0
--local_ip_for_outbound_sockets=127.2.2.2
--webserver_interface=127.2.2.2
--webserver_port=0
--tserver_master_addrs=127.2.2.62:39059
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/krb5kdc/oryx_127.2.2.2.keytab
--principal=oryx/127.2.2.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:45229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:32:29.431351 8374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:29.431653 8374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:29.431740 8374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:29.436012 8374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:29.436082 8374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:29.436285 8374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.2
I20260619 14:32:29.441926 8374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:45229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/krb5kdc/oryx_127.2.2.2.keytab
--principal=oryx/127.2.2.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.2.2
--webserver_port=0
--tserver_master_addrs=127.2.2.62:39059
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.2
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:32:29.443079 8374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:29.444130 8374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:32:29.453979 8382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:29.453979 8380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:29.454222 8374 server_base.cc:1034] running on GCE node
W20260619 14:32:29.454367 8379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:29.454835 8374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:29.455557 8374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:29.456777 8374 hybrid_clock.cc:648] HybridClock initialized: now 1781879549456738 us; error 59 us; skew 500 ppm
Jun 19 14:32:29 dist-test-slave-70rm krb5kdc[8133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879549, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.2.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:32:29.459865 8374 init.cc:377] Logged in from keytab as oryx/127.2.2.2@KRBTEST.COM (short username oryx)
I20260619 14:32:29.460856 8374 webserver.cc:460] Webserver started at http://127.2.2.2:34185/ using document root <none> and password file <none>
I20260619 14:32:29.461460 8374 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:29.461570 8374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:29.461810 8374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:32:29.464077 8374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-1/data/instance:
uuid: "3cc7f187edfc4edabbb95a0f1bdce56a"
format_stamp: "Formatted at 2026-06-19 14:32:29 on dist-test-slave-70rm"
server_key: "ac7890c3135cb9d16e1355506936326d"
server_key_iv: "772972fdb895fd2cb0e018aec1422fcd"
server_key_version: "encryptionkey@0"
I20260619 14:32:29.464664 8374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-1/wal/instance:
uuid: "3cc7f187edfc4edabbb95a0f1bdce56a"
format_stamp: "Formatted at 2026-06-19 14:32:29 on dist-test-slave-70rm"
server_key: "ac7890c3135cb9d16e1355506936326d"
server_key_iv: "772972fdb895fd2cb0e018aec1422fcd"
server_key_version: "encryptionkey@0"
I20260619 14:32:29.468679 8374 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:32:29.471433 8389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:29.472980 8374 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 14:32:29.473115 8374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-1/wal
uuid: "3cc7f187edfc4edabbb95a0f1bdce56a"
format_stamp: "Formatted at 2026-06-19 14:32:29 on dist-test-slave-70rm"
server_key: "ac7890c3135cb9d16e1355506936326d"
server_key_iv: "772972fdb895fd2cb0e018aec1422fcd"
server_key_version: "encryptionkey@0"
I20260619 14:32:29.473213 8374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:29.488070 8374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:29.490937 8374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:32:29.491098 8374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:29.491613 8374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:32:29.492396 8374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:32:29.492442 8374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:29.492482 8374 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:32:29.492503 8374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:29.501216 8374 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.2:34039
I20260619 14:32:29.501274 8502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.2:34039 every 8 connection(s)
I20260619 14:32:29.502492 8374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
I20260619 14:32:29.505119 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 8374
I20260619 14:32:29.505252 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-1/wal/instance
I20260619 14:32:29.505573 2056 external_mini_cluster.cc:1428] Setting key 8652bae9397693fb44397f7a431c1847
Jun 19 14:32:29 dist-test-slave-70rm krb5kdc[8133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879549, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.2.2@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.2.2.3@KRBTEST.COM; defaulting to no policy
I20260619 14:32:29.516746 8217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:29.504357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:45713 (local address 127.2.2.62:39059)
0619 14:32:29.504528 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:29.504540 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:29.505549 (+ 1009us) server_negotiation.cc:407] Connection header received
0619 14:32:29.507134 (+ 1585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:29.507137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:29.507187 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:29.507288 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:29.508469 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:29.508994 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:29.510171 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:29.510424 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:29.510539 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:29.513047 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:29.513051 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:29.513053 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:29.513079 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:29.514656 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:29.515185 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:29.515189 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:29.515191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:29.515272 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:29.515611 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:29.515616 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:29.515618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:29.515816 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:29.515925 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:29.516443 (+ 518us) server_negotiation.cc:299] Negotiation successful
0619 14:32:29.516572 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.2.3@KRBTEST.COM" created.
I20260619 14:32:29.517791 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:29.504765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:39059 (local address 127.2.2.2:45713)
0619 14:32:29.505395 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:29.505439 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:29.506922 (+ 1483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:29.507468 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:29.507475 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:29.507840 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:29.508328 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:29.508342 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:29.509122 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:29.509125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:29.510029 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:29.510037 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:29.510525 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:29.510583 (+ 58us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:29.510773 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:29.511382 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:29.511399 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:29.512859 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:29.514783 (+ 1924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:29.514790 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:29.514809 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:29.515075 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:29.515366 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:29.515369 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:29.515371 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:29.515516 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:29.515920 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:29.515927 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:29.516195 (+ 268us) client_negotiation.cc:769] Sending connection context
0619 14:32:29.516403 (+ 208us) client_negotiation.cc:240] Negotiation successful
0619 14:32:29.516743 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":168,"threads_started":1}
I20260619 14:32:29.519080 8503 heartbeater.cc:347] Connected to a master server at 127.2.2.62:39059
I20260619 14:32:29.519371 8503 heartbeater.cc:464] Registering TS with master...
I20260619 14:32:29.519868 8503 heartbeater.cc:511] Master 127.2.2.62:39059 requested a full tablet report, sending...
I20260619 14:32:29.520913 8174 ts_manager.cc:194] Registered new tserver with Master: 3cc7f187edfc4edabbb95a0f1bdce56a (127.2.2.2:34039)
I20260619 14:32:29.521502 8174 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.2.2@KRBTEST.COM'} at 127.2.2.2:45713
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.2.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/krb5kdc/oryx_127.2.2.3.keytab.
Entry for principal oryx/127.2.2.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/krb5kdc/oryx_127.2.2.3.keytab.
I20260619 14:32:29.533202 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.3:0
--local_ip_for_outbound_sockets=127.2.2.3
--webserver_interface=127.2.2.3
--webserver_port=0
--tserver_master_addrs=127.2.2.62:39059
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/krb5kdc/oryx_127.2.2.3.keytab
--principal=oryx/127.2.2.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:45229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:32:29.652547 8508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:29.652827 8508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:29.652870 8508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:29.657248 8508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:29.657315 8508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:29.657375 8508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.3
I20260619 14:32:29.662866 8508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:45229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/krb5kdc/oryx_127.2.2.3.keytab
--principal=oryx/127.2.2.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.2.3
--webserver_port=0
--tserver_master_addrs=127.2.2.62:39059
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.3
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:32:29.663904 8508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:29.664877 8508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:32:29.675426 8513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:29.675618 8508 server_base.cc:1034] running on GCE node
W20260619 14:32:29.675428 8516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:29.675426 8514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:29.676290 8508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:29.676999 8508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:29.678233 8508 hybrid_clock.cc:648] HybridClock initialized: now 1781879549678214 us; error 35 us; skew 500 ppm
Jun 19 14:32:29 dist-test-slave-70rm krb5kdc[8133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879549, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.2.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:32:29.681371 8508 init.cc:377] Logged in from keytab as oryx/127.2.2.3@KRBTEST.COM (short username oryx)
I20260619 14:32:29.682523 8508 webserver.cc:460] Webserver started at http://127.2.2.3:41245/ using document root <none> and password file <none>
I20260619 14:32:29.683148 8508 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:29.683257 8508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:29.683516 8508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:32:29.685808 8508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-2/data/instance:
uuid: "c0f8745082c44f37a47d562686cf2780"
format_stamp: "Formatted at 2026-06-19 14:32:29 on dist-test-slave-70rm"
server_key: "2131872fe254872417e2d0c4d2b6aa76"
server_key_iv: "984966f9fa528722dde61a9010d54eae"
server_key_version: "encryptionkey@0"
I20260619 14:32:29.686489 8508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-2/wal/instance:
uuid: "c0f8745082c44f37a47d562686cf2780"
format_stamp: "Formatted at 2026-06-19 14:32:29 on dist-test-slave-70rm"
server_key: "2131872fe254872417e2d0c4d2b6aa76"
server_key_iv: "984966f9fa528722dde61a9010d54eae"
server_key_version: "encryptionkey@0"
I20260619 14:32:29.690928 8508 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:32:29.693815 8523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:29.695335 8508 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:32:29.695510 8508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-2/wal
uuid: "c0f8745082c44f37a47d562686cf2780"
format_stamp: "Formatted at 2026-06-19 14:32:29 on dist-test-slave-70rm"
server_key: "2131872fe254872417e2d0c4d2b6aa76"
server_key_iv: "984966f9fa528722dde61a9010d54eae"
server_key_version: "encryptionkey@0"
I20260619 14:32:29.695655 8508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:29.718236 8508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:29.722616 8508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:32:29.722849 8508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:29.723587 8508 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:32:29.724712 8508 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:32:29.724788 8508 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:29.724872 8508 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:32:29.724905 8508 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:29.736676 8508 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.3:41487
I20260619 14:32:29.736691 8636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.3:41487 every 8 connection(s)
I20260619 14:32:29.737651 8508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
I20260619 14:32:29.740794 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 8508
I20260619 14:32:29.740921 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-2/wal/instance
I20260619 14:32:29.741194 2056 external_mini_cluster.cc:1428] Setting key 0b1bad05c87ead0e3dc8faeef89c805c
Jun 19 14:32:29 dist-test-slave-70rm krb5kdc[8133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879549, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.2.3@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM
I20260619 14:32:29.755412 8217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:29.739609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.3:44397 (local address 127.2.2.62:39059)
0619 14:32:29.739754 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:29.739758 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:29.740739 (+ 981us) server_negotiation.cc:407] Connection header received
0619 14:32:29.742378 (+ 1639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:29.742382 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:29.742443 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:29.742532 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:29.744107 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:29.744938 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:29.747177 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:29.747600 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:29.747734 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:29.751011 (+ 3277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:29.751016 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:29.751021 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:29.751060 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:29.752847 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:29.753418 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:29.753422 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:29.753424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:29.753515 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:29.753900 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:29.753903 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:29.753905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:29.754237 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:29.754339 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:29.755087 (+ 748us) server_negotiation.cc:299] Negotiation successful
0619 14:32:29.755243 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 14:32:29.756325 8639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:29.740057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:39059 (local address 127.2.2.3:44397)
0619 14:32:29.740565 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:29.740609 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:29.742110 (+ 1501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:29.742842 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:29.742852 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:29.743364 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:29.743946 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:29.743961 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:29.745116 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:29.745119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:29.747014 (+ 1895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:29.747041 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:29.747732 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:29.747795 (+ 63us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:29.748106 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:29.748943 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:29.748966 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:29.750790 (+ 1824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:29.753006 (+ 2216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:29.753012 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:29.753026 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:29.753299 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:29.753624 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:29.753631 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:29.753633 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:29.753792 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:29.754362 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:29.754371 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:29.754775 (+ 404us) client_negotiation.cc:769] Sending connection context
0619 14:32:29.755020 (+ 245us) client_negotiation.cc:240] Negotiation successful
0619 14:32:29.755417 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":122,"threads_started":1}
I20260619 14:32:29.757481 8637 heartbeater.cc:347] Connected to a master server at 127.2.2.62:39059
I20260619 14:32:29.757757 8637 heartbeater.cc:464] Registering TS with master...
I20260619 14:32:29.758358 8637 heartbeater.cc:511] Master 127.2.2.62:39059 requested a full tablet report, sending...
I20260619 14:32:29.759442 8174 ts_manager.cc:194] Registered new tserver with Master: c0f8745082c44f37a47d562686cf2780 (127.2.2.3:41487)
I20260619 14:32:29.759986 8174 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.2.3@KRBTEST.COM'} at 127.2.2.3:44397
I20260619 14:32:29.767416 2056 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.2.2.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.2.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.2.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/krb5kdc/oryx_127.2.2.61.keytab.
Entry for principal oryx/127.2.2.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/krb5kdc/oryx_127.2.2.61.keytab.
I20260619 14:32:29.792917 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.61:36895
--webserver_interface=127.2.2.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/krb5kdc/oryx_127.2.2.61.keytab
--principal=oryx/127.2.2.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:45229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.2.62:39059,127.2.2.61:36895
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:32:29.905125 8642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:29.905448 8642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:29.905490 8642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:29.909781 8642 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:32:29.909863 8642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:29.909884 8642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:29.909902 8642 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:32:29.909917 8642 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:32:29.915556 8642 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:45229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.2.62:39059,127.2.2.61:36895
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/krb5kdc/oryx_127.2.2.61.keytab
--principal=oryx/127.2.2.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.61:36895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.2.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
W20260619 14:32:29.915829 8642 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.2.2.62:39059,127.2.2.61:36895'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260619 14:32:29.916766 8642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:29.917846 8642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:32:29.926438 8650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:29.926406 8648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:29.926414 8647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:29.926590 8642 server_base.cc:1034] running on GCE node
I20260619 14:32:29.927328 8642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:29.928439 8642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:29.929630 8642 hybrid_clock.cc:648] HybridClock initialized: now 1781879549929608 us; error 29 us; skew 500 ppm
Jun 19 14:32:29 dist-test-slave-70rm krb5kdc[8133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879549, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.2.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:32:29.932508 8642 init.cc:377] Logged in from keytab as oryx/127.2.2.61@KRBTEST.COM (short username oryx)
I20260619 14:32:29.933441 8642 webserver.cc:460] Webserver started at http://127.2.2.61:41809/ using document root <none> and password file <none>
I20260619 14:32:29.934015 8642 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:29.934108 8642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:29.934341 8642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:32:29.936385 8642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-1/data/instance:
uuid: "e36baac4eca2403197084187e1360c08"
format_stamp: "Formatted at 2026-06-19 14:32:29 on dist-test-slave-70rm"
server_key: "72c616c37a3c063bef5ca62fbbe8687b"
server_key_iv: "45f62aba71228ec5fd93c527e182e286"
server_key_version: "encryptionkey@0"
I20260619 14:32:29.936936 8642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-1/wal/instance:
uuid: "e36baac4eca2403197084187e1360c08"
format_stamp: "Formatted at 2026-06-19 14:32:29 on dist-test-slave-70rm"
server_key: "72c616c37a3c063bef5ca62fbbe8687b"
server_key_iv: "45f62aba71228ec5fd93c527e182e286"
server_key_version: "encryptionkey@0"
I20260619 14:32:29.940850 8642 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260619 14:32:29.943549 8657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:29.944823 8642 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:32:29.944993 8642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-1/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-1/wal
uuid: "e36baac4eca2403197084187e1360c08"
format_stamp: "Formatted at 2026-06-19 14:32:29 on dist-test-slave-70rm"
server_key: "72c616c37a3c063bef5ca62fbbe8687b"
server_key_iv: "45f62aba71228ec5fd93c527e182e286"
server_key_version: "encryptionkey@0"
I20260619 14:32:29.945125 8642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:29.995776 8642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:29.999073 8642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:32:29.999349 8642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:30.007200 8642 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.61:36895
I20260619 14:32:30.007539 8709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.61:36895 every 8 connection(s)
I20260619 14:32:30.008255 8642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-1/data/info.pb
I20260619 14:32:30.009770 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 8642
I20260619 14:32:30.011747 8710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:30.016353 8710 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.2.62" port: 39059 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:32:30 dist-test-slave-70rm krb5kdc[8133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.2.61@KRBTEST.COM
Jun 19 14:32:30 dist-test-slave-70rm krb5kdc[8133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879549, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.2.61@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM
I20260619 14:32:30.025257 8713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:30.011356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37354 (local address 127.2.2.61:36895)
0619 14:32:30.011867 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:30.011890 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:30.011945 (+ 55us) server_negotiation.cc:407] Connection header received
0619 14:32:30.012638 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:30.012654 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:30.012982 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:30.013337 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:30.014209 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.015202 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.016046 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.016531 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.016714 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:30.019789 (+ 3075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:30.019795 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:30.019804 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:30.019833 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:30.021741 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:30.022332 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:30.022355 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:30.022363 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:30.022508 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:30.022795 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:30.022798 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:30.022800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:30.023263 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:30.023418 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:30.023826 (+ 408us) server_negotiation.cc:299] Negotiation successful
0619 14:32:30.024179 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":152,"threads_started":1}
I20260619 14:32:30.027952 8714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:30.018195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:39059 (local address 127.0.0.1:37750)
0619 14:32:30.018574 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:30.018624 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:30.018743 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:30.019110 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:30.019115 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:30.019515 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:30.019646 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:30.019651 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.020582 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.020586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:30.021242 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:30.021250 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.021692 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.021707 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:30.021824 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:30.022426 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:30.022439 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:30.024105 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:30.026260 (+ 2155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:30.026267 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:30.026271 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:30.026524 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:30.026795 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:30.026798 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:30.026800 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:30.026914 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:30.027341 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:30.027348 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:30.027458 (+ 110us) client_negotiation.cc:769] Sending connection context
0619 14:32:30.027613 (+ 155us) client_negotiation.cc:240] Negotiation successful
0619 14:32:30.027796 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":122,"threads_started":1}
I20260619 14:32:30.027984 8217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:30.018274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37750 (local address 127.2.2.62:39059)
0619 14:32:30.018447 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:30.018455 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:30.018712 (+ 257us) server_negotiation.cc:407] Connection header received
0619 14:32:30.018848 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:30.018853 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:30.018915 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:30.019038 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:30.019775 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.020441 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.021368 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.021595 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.021707 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:30.024240 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:30.024244 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:30.024246 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:30.024272 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:30.026162 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:30.026625 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:30.026628 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:30.026629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:30.026704 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:30.027025 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:30.027027 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:30.027028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:30.027233 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:30.027362 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:30.027714 (+ 352us) server_negotiation.cc:299] Negotiation successful
0619 14:32:30.027843 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 14:32:30.029045 8710 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.2.61" port: 36895 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:32:30 dist-test-slave-70rm krb5kdc[8133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879549, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.2.61@KRBTEST.COM for oryx/127.2.2.61@KRBTEST.COM
I20260619 14:32:30.037284 8713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:30.029634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37366 (local address 127.2.2.61:36895)
0619 14:32:30.029727 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:30.029733 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:30.029746 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:32:30.029851 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:30.029855 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:30.029892 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:30.029974 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:30.030359 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.031318 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.032069 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.032301 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.032408 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:30.034405 (+ 1997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:30.034409 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:30.034414 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:30.034438 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:30.035724 (+ 1286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:30.036159 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:30.036165 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:30.036167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:30.036240 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:30.036505 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:30.036508 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:30.036510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:30.036677 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:30.036779 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:30.037012 (+ 233us) server_negotiation.cc:299] Negotiation successful
0619 14:32:30.037153 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":40}
I20260619 14:32:30.037284 8714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:30.029564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.61:36895 (local address 127.0.0.1:37366)
0619 14:32:30.029694 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:30.029708 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:30.029781 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:30.029980 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:30.029983 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:30.030170 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:30.030257 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:30.030262 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.031433 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.031437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:30.031958 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:30.031965 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.032392 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.032405 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:30.032489 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:30.032962 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:30.032973 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:30.034276 (+ 1303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:30.035860 (+ 1584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:30.035864 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:30.035866 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:30.036067 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:30.036328 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:30.036333 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:30.036335 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:30.036416 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:30.036768 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:30.036770 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:30.036845 (+ 75us) client_negotiation.cc:769] Sending connection context
0619 14:32:30.036954 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 14:32:30.037101 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 14:32:30.040298 8710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08: Bootstrap starting.
I20260619 14:32:30.042680 8710 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:30.043391 8710 log.cc:824] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08: Log is configured to *not* fsync() on all Append() calls
I20260619 14:32:30.045228 8710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08: No bootstrap required, opened a new log
I20260619 14:32:30.048148 8710 raft_consensus.cc:348] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf27644a9d194a898a1d93d6821e8b38" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 39059 } } peers { permanent_uuid: "e36baac4eca2403197084187e1360c08" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 36895 } }
I20260619 14:32:30.048385 8710 raft_consensus.cc:374] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:30.048451 8710 raft_consensus.cc:729] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e36baac4eca2403197084187e1360c08, State: Initialized, Role: FOLLOWER
I20260619 14:32:30.048863 8710 consensus_queue.cc:260] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf27644a9d194a898a1d93d6821e8b38" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 39059 } } peers { permanent_uuid: "e36baac4eca2403197084187e1360c08" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 36895 } }
I20260619 14:32:30.049377 8716 sys_catalog.cc:455] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf27644a9d194a898a1d93d6821e8b38" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 39059 } } peers { permanent_uuid: "e36baac4eca2403197084187e1360c08" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 36895 } } }
I20260619 14:32:30.049520 8716 sys_catalog.cc:458] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:32:30.050086 8710 sys_catalog.cc:564] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 14:32:30.054955 8727 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:32:30.055074 8727 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:32:30.057286 8642 master_runner.cc:428] Detected that this master e36baac4eca2403197084187e1360c08 is joining an existing cluster
I20260619 14:32:30.057358 8642 master_runner.cc:432] Initiating AddMaster RPC to add 127.2.2.61:36895
I20260619 14:32:30.068482 8734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:30.059812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.61:36895 (local address 127.0.0.1:37378)
0619 14:32:30.060422 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:30.060435 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:30.060521 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:30.060846 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:30.060852 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:30.061033 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:30.061148 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:30.061156 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.061867 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.061870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:30.062786 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:30.062796 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.063372 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.063390 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:30.063491 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:30.064542 (+ 1051us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:30.064555 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:30.064974 (+ 419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:30.066833 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:30.066839 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:30.066842 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:30.067074 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:30.067387 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:30.067390 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:30.067392 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:30.067466 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:30.067903 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:30.067906 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:30.068001 (+ 95us) client_negotiation.cc:769] Sending connection context
0619 14:32:30.068135 (+ 134us) client_negotiation.cc:240] Negotiation successful
0619 14:32:30.068283 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"mutex_wait_us":3,"thread_start_us":91,"threads_started":1}
I20260619 14:32:30.068764 8217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:30.059865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37758 (local address 127.2.2.62:39059)
0619 14:32:30.059993 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:30.059998 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:30.060313 (+ 315us) server_negotiation.cc:407] Connection header received
0619 14:32:30.060436 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:30.060440 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:30.060498 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:30.060603 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:30.061187 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.062019 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.063135 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.063399 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.063491 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:30.064995 (+ 1504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:30.065001 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:30.065004 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:30.065031 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:30.066773 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:30.067326 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:30.067330 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:30.067332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:30.067417 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:30.067731 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:30.067734 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:30.067736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:30.067951 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:30.068064 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:30.068343 (+ 279us) server_negotiation.cc:299] Negotiation successful
0619 14:32:30.068456 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 14:32:30.068837 8733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:30.059801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:39059 (local address 127.0.0.1:37758)
0619 14:32:30.060203 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:30.060218 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:30.060335 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:30.060616 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:30.060618 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:30.060873 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:30.061021 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:30.061028 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.062176 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.062179 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:30.063015 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:30.063021 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.063605 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.063622 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:30.063730 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:30.064238 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:30.064255 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:30.064840 (+ 585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:30.066903 (+ 2063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:30.066906 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:30.066908 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:30.067207 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:30.067536 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:30.067539 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:30.067541 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:30.067637 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:30.068059 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:30.068062 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:30.068146 (+ 84us) client_negotiation.cc:769] Sending connection context
0619 14:32:30.068293 (+ 147us) client_negotiation.cc:240] Negotiation successful
0619 14:32:30.068421 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"spinlock_wait_cycles":1792,"thread_start_us":100,"threads_started":1}
I20260619 14:32:30.069504 8713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:30.059920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37378 (local address 127.2.2.61:36895)
0619 14:32:30.060014 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:30.060018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:30.060507 (+ 489us) server_negotiation.cc:407] Connection header received
0619 14:32:30.060625 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:30.060627 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:30.060665 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:30.060715 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:30.061230 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.061763 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.062910 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.063257 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.063369 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:30.065104 (+ 1735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:30.065108 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:30.065111 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:30.065130 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:30.066726 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:30.067172 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:30.067175 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:30.067177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:30.067278 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:30.067547 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:30.067551 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:30.067554 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:30.067801 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:30.067901 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:30.068180 (+ 279us) server_negotiation.cc:299] Negotiation successful
0619 14:32:30.069377 (+ 1197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1076,"server-negotiator.queue_time_us":35,"spinlock_wait_cycles":40192}
Jun 19 14:32:30 dist-test-slave-70rm krb5kdc[8133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879548, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.2.62@KRBTEST.COM for oryx/127.2.2.61@KRBTEST.COM
I20260619 14:32:30.083202 8735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:30.073322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.61:36895 (local address 127.0.0.1:37388)
0619 14:32:30.073689 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:30.073731 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:30.073886 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:30.074239 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:30.074245 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:30.074629 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:30.074754 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:30.074761 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.075485 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.075491 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:30.076245 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:30.076256 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.076744 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.076764 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:30.076888 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:30.077575 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:30.077587 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:30.079264 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:30.081111 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:30.081122 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:30.081129 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:30.081448 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:30.081772 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:30.081775 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:30.081777 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:30.081876 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:30.082458 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:30.082466 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:30.082612 (+ 146us) client_negotiation.cc:769] Sending connection context
0619 14:32:30.082764 (+ 152us) client_negotiation.cc:240] Negotiation successful
0619 14:32:30.082981 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":109,"threads_started":1}
I20260619 14:32:30.083243 8713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:30.073329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37388 (local address 127.2.2.61:36895)
0619 14:32:30.073435 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:30.073446 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:30.073808 (+ 362us) server_negotiation.cc:407] Connection header received
0619 14:32:30.074024 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:30.074027 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:30.074082 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:30.074195 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:30.074875 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.075374 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.076365 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.076641 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.076747 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:30.079408 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:30.079412 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:30.079414 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:30.079436 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:30.080984 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:30.081559 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:30.081566 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:30.081568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:30.081670 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:30.081973 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:30.081979 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:30.081983 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:30.082343 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:30.082450 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:30.082843 (+ 393us) server_negotiation.cc:299] Negotiation successful
0619 14:32:30.082983 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 14:32:30.085412 8174 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.2.2.61:36895: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e36baac4eca2403197084187e1360c08"
member_type: NON_VOTER
last_known_addr {
host: "127.2.2.61"
port: 36895
}
attrs {
promote: true
}
}
dest_uuid: "cf27644a9d194a898a1d93d6821e8b38"
cas_config_opid_index: -1
I20260619 14:32:30.086256 8174 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf27644a9d194a898a1d93d6821e8b38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cf27644a9d194a898a1d93d6821e8b38" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 39059 } } peers { permanent_uuid: "e36baac4eca2403197084187e1360c08" member_type: NON_VOTER last_known_addr { host: "127.2.2.61" port: 36895 } attrs { promote: true } }
WARNING: no policy specified for oryx/127.2.2.60@KRBTEST.COM; defaulting to no policy
I20260619 14:32:30.088418 8739 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cf27644a9d194a898a1d93d6821e8b38 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER e36baac4eca2403197084187e1360c08 (127.2.2.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cf27644a9d194a898a1d93d6821e8b38" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 39059 } } peers { permanent_uuid: "e36baac4eca2403197084187e1360c08" member_type: NON_VOTER last_known_addr { host: "127.2.2.61" port: 36895 } attrs { promote: true } } }
I20260619 14:32:30.089334 8684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:30.090835 8739 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.2.2.61:36895
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.2.60@KRBTEST.COM" created.
I20260619 14:32:30.091532 8684 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [term 1 FOLLOWER]: Refusing update from remote peer cf27644a9d194a898a1d93d6821e8b38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260619 14:32:30.091768 8738 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf27644a9d194a898a1d93d6821e8b38 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "cf27644a9d194a898a1d93d6821e8b38" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cf27644a9d194a898a1d93d6821e8b38" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 39059 } } peers { permanent_uuid: "e36baac4eca2403197084187e1360c08" member_type: NON_VOTER last_known_addr { host: "127.2.2.61" port: 36895 } attrs { promote: true } } }
I20260619 14:32:30.091869 8738 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf27644a9d194a898a1d93d6821e8b38 [sys.catalog]: This master's current role is: LEADER
I20260619 14:32:30.092361 8738 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cf27644a9d194a898a1d93d6821e8b38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e36baac4eca2403197084187e1360c08" member_type: NON_VOTER last_known_addr { host: "127.2.2.61" port: 36895 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:32:30.092685 8642 master.cc:537] Master@127.2.2.61:36895 shutting down...
W20260619 14:32:30.095275 8162 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260619 14:32:30.097527 8642 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:32:30.097844 8642 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:32:30.097967 8642 tablet_replica.cc:330] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08: stopping tablet replica
I20260619 14:32:30.103045 8735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:30.102721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.61:36895 (local address 127.0.0.1:37400)
0619 14:32:30.102866 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:30.102948 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.61:36895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 14:32:30.103334 8162 consensus_peers.cc:487] T 00000000000000000000000000000000 P cf27644a9d194a898a1d93d6821e8b38 -> Peer e36baac4eca2403197084187e1360c08 (127.2.2.61:36895): Couldn't send request to peer e36baac4eca2403197084187e1360c08. Status: Network error: Client connection negotiation failed: client connection to 127.2.2.61:36895: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.2.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/krb5kdc/oryx_127.2.2.60.keytab.
Entry for principal oryx/127.2.2.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/krb5kdc/oryx_127.2.2.60.keytab.
I20260619 14:32:30.110332 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.60:45719
--webserver_interface=127.2.2.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/krb5kdc/oryx_127.2.2.60.keytab
--principal=oryx/127.2.2.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:45229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.2.62:39059,127.2.2.61:36895,127.2.2.60:45719
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:32:30.113535 8642 master.cc:552] Master@127.2.2.61:36895 shutdown complete.
I20260619 14:32:30.113620 8642 master_runner.cc:305] Clearing existing system tablet
I20260619 14:32:30.113860 8642 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:30.113929 8642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:30.115209 8642 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 14:32:30.116071 8745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:30.116281 8642 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:30.116407 8642 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 14:32:30.116446 8642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-1/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-1/wal
uuid: "e36baac4eca2403197084187e1360c08"
format_stamp: "Formatted at 2026-06-19 14:32:29 on dist-test-slave-70rm"
server_key: "72c616c37a3c063bef5ca62fbbe8687b"
server_key_iv: "45f62aba71228ec5fd93c527e182e286"
server_key_version: "encryptionkey@0"
I20260619 14:32:30.119508 8642 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:32:30.121464 8642 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 14:32:30.121574 8642 log.cc:1196] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08: Deleting WAL directory at /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260619 14:32:30.121847 8642 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08: Deleting consensus metadata
I20260619 14:32:30.122032 8642 master_runner.cc:315] Copying system tablet from 127.2.2.62:39059
I20260619 14:32:30.123251 8642 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08: tablet copy: Beginning tablet copy session from remote peer at address 127.2.2.62:39059
I20260619 14:32:30.131450 8217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:30.123918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37772 (local address 127.2.2.62:39059)
0619 14:32:30.124050 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:30.124055 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:30.124272 (+ 217us) server_negotiation.cc:407] Connection header received
0619 14:32:30.124450 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:30.124453 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:30.124508 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:30.124598 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:30.125121 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.125735 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.126810 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.127042 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.127155 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:30.128311 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:30.128315 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:30.128318 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:30.128346 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:30.129784 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:30.130254 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:30.130259 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:30.130260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:30.130340 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:30.130628 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:30.130632 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:30.130633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:30.130821 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:30.130926 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:30.131196 (+ 270us) server_negotiation.cc:299] Negotiation successful
0619 14:32:30.131304 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 14:32:30.131464 8751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:30.123830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:39059 (local address 127.0.0.1:37772)
0619 14:32:30.124172 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:30.124185 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:30.124332 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:30.124618 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:30.124621 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:30.124848 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:30.124993 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:30.125001 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.125846 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.125852 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:30.126698 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:30.126708 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.127184 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.127205 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:30.127302 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:30.127843 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:30.127855 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:30.128167 (+ 312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:30.129911 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:30.129914 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:30.129916 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:30.130114 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:30.130433 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:30.130437 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:30.130439 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:30.130523 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:30.130928 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:30.130931 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:30.131030 (+ 99us) client_negotiation.cc:769] Sending connection context
0619 14:32:30.131133 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 14:32:30.131274 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":89,"threads_started":1}
I20260619 14:32:30.132283 8198 tablet_copy_service.cc:140] P cf27644a9d194a898a1d93d6821e8b38: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer e36baac4eca2403197084187e1360c08 ({username='oryx', principal='oryx/127.2.2.61@KRBTEST.COM'} at 127.0.0.1:37772)
I20260619 14:32:30.132460 8198 tablet_copy_service.cc:161] P cf27644a9d194a898a1d93d6821e8b38: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer e36baac4eca2403197084187e1360c08 at {username='oryx', principal='oryx/127.2.2.61@KRBTEST.COM'} at 127.0.0.1:37772: session id = e36baac4eca2403197084187e1360c08-00000000000000000000000000000000
I20260619 14:32:30.134053 8198 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P cf27644a9d194a898a1d93d6821e8b38: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 14:32:30.135551 8642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 14:32:30.137388 8642 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08: tablet copy: Starting download of 0 data blocks...
I20260619 14:32:30.137598 8642 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08: tablet copy: Starting download of 1 WAL segments...
I20260619 14:32:30.139216 8642 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 14:32:30.140750 8198 tablet_copy_service.cc:342] P cf27644a9d194a898a1d93d6821e8b38: Request end of tablet copy session e36baac4eca2403197084187e1360c08-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.2.2.61@KRBTEST.COM'} at 127.0.0.1:37772
I20260619 14:32:30.140868 8198 tablet_copy_service.cc:433] P cf27644a9d194a898a1d93d6821e8b38: ending tablet copy session e36baac4eca2403197084187e1360c08-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer e36baac4eca2403197084187e1360c08
I20260619 14:32:30.147612 8642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:32:30.149601 8752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:30.149605 8755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:30.149972 8642 server_base.cc:1034] running on GCE node
W20260619 14:32:30.150228 8753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:30.150454 8642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:30.150816 8642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:30.151989 8642 hybrid_clock.cc:648] HybridClock initialized: now 1781879550151993 us; error 43 us; skew 500 ppm
Jun 19 14:32:30 dist-test-slave-70rm krb5kdc[8133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879550, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.2.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:32:30.153538 8642 init.cc:377] Logged in from keytab as oryx/127.2.2.61@KRBTEST.COM (short username oryx)
I20260619 14:32:30.154039 8642 webserver.cc:460] Webserver started at http://127.2.2.61:33043/ using document root <none> and password file <none>
I20260619 14:32:30.154271 8642 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:30.154328 8642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:30.155462 8642 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 14:32:30.156486 8762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:30.156738 8642 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 14:32:30.156854 8642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-1/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-1/wal
uuid: "e36baac4eca2403197084187e1360c08"
format_stamp: "Formatted at 2026-06-19 14:32:29 on dist-test-slave-70rm"
server_key: "72c616c37a3c063bef5ca62fbbe8687b"
server_key_iv: "45f62aba71228ec5fd93c527e182e286"
server_key_version: "encryptionkey@0"
I20260619 14:32:30.156936 8642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:30.184134 8642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:30.186322 8642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:30.191252 8642 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.61:36895
I20260619 14:32:30.191805 8642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-1/data/info.pb
I20260619 14:32:30.193378 8815 sys_catalog.cc:263] Verifying existing consensus state
I20260619 14:32:30.194314 8815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08: Bootstrap starting.
I20260619 14:32:30.198231 8814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.61:36895 every 8 connection(s)
I20260619 14:32:30.202833 8815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:32:30.203177 8815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08: Bootstrap complete.
I20260619 14:32:30.203711 8815 raft_consensus.cc:348] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cf27644a9d194a898a1d93d6821e8b38" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 39059 } } peers { permanent_uuid: "e36baac4eca2403197084187e1360c08" member_type: NON_VOTER last_known_addr { host: "127.2.2.61" port: 36895 } attrs { promote: true } }
I20260619 14:32:30.203848 8815 raft_consensus.cc:729] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e36baac4eca2403197084187e1360c08, State: Initialized, Role: LEARNER
I20260619 14:32:30.203958 8815 consensus_queue.cc:260] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cf27644a9d194a898a1d93d6821e8b38" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 39059 } } peers { permanent_uuid: "e36baac4eca2403197084187e1360c08" member_type: NON_VOTER last_known_addr { host: "127.2.2.61" port: 36895 } attrs { promote: true } }
I20260619 14:32:30.204319 8818 sys_catalog.cc:455] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cf27644a9d194a898a1d93d6821e8b38" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 39059 } } peers { permanent_uuid: "e36baac4eca2403197084187e1360c08" member_type: NON_VOTER last_known_addr { host: "127.2.2.61" port: 36895 } attrs { promote: true } } }
I20260619 14:32:30.204476 8818 sys_catalog.cc:458] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [sys.catalog]: This master's current role is: LEARNER
I20260619 14:32:30.204496 8815 sys_catalog.cc:564] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:32:30.211103 8829 catalog_manager.cc:1259] Loaded cluster ID: 96382bd00e6e4b0792b1bfa035e796dc
I20260619 14:32:30.211192 8829 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08: loading cluster ID for follower catalog manager: success
I20260619 14:32:30.212723 8829 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08: acquiring CA information for follower catalog manager: success
I20260619 14:32:30.213858 8829 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 14:32:30.266496 8743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:30.266780 8743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:30.266851 8743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:30.270933 8743 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:32:30.270998 8743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:30.271018 8743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:30.271034 8743 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:32:30.271087 8743 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:32:30.276551 8743 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:45229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.2.62:39059,127.2.2.61:36895,127.2.2.60:45719
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/krb5kdc/oryx_127.2.2.60.keytab
--principal=oryx/127.2.2.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.60:45719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.2.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:32:30.277769 8743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:30.278911 8743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:32:30.288381 8835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:30.288570 8743 server_base.cc:1034] running on GCE node
W20260619 14:32:30.288383 8834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:30.288383 8837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:30.289337 8743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:30.290563 8743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:30.291759 8743 hybrid_clock.cc:648] HybridClock initialized: now 1781879550291719 us; error 51 us; skew 500 ppm
Jun 19 14:32:30 dist-test-slave-70rm krb5kdc[8133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879550, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.2.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:32:30.296164 8743 init.cc:377] Logged in from keytab as oryx/127.2.2.60@KRBTEST.COM (short username oryx)
I20260619 14:32:30.297410 8743 webserver.cc:460] Webserver started at http://127.2.2.60:37047/ using document root <none> and password file <none>
I20260619 14:32:30.298233 8743 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:30.298328 8743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:30.298575 8743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:32:30.301343 8743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-2/data/instance:
uuid: "03d127796b8d4c3286344b40cff82093"
format_stamp: "Formatted at 2026-06-19 14:32:30 on dist-test-slave-70rm"
server_key: "04e697cab2bdc8560576834a6066f799"
server_key_iv: "d29990d4c698c7f5563489ab0a59160d"
server_key_version: "encryptionkey@0"
I20260619 14:32:30.301877 8743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-2/wal/instance:
uuid: "03d127796b8d4c3286344b40cff82093"
format_stamp: "Formatted at 2026-06-19 14:32:30 on dist-test-slave-70rm"
server_key: "04e697cab2bdc8560576834a6066f799"
server_key_iv: "d29990d4c698c7f5563489ab0a59160d"
server_key_version: "encryptionkey@0"
I20260619 14:32:30.305490 8369 heartbeater.cc:503] Master 127.2.2.62:39059 was elected leader, sending a full tablet report...
I20260619 14:32:30.306511 8743 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 14:32:30.309057 8844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:30.310407 8743 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:32:30.310552 8743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-2/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-2/wal
uuid: "03d127796b8d4c3286344b40cff82093"
format_stamp: "Formatted at 2026-06-19 14:32:30 on dist-test-slave-70rm"
server_key: "04e697cab2bdc8560576834a6066f799"
server_key_iv: "d29990d4c698c7f5563489ab0a59160d"
server_key_version: "encryptionkey@0"
I20260619 14:32:30.310672 8743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:30.321693 8743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:30.325158 8743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:32:30.325407 8743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:30.333318 8743 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.60:45719
I20260619 14:32:30.333592 8896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.60:45719 every 8 connection(s)
I20260619 14:32:30.334398 8743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-2/data/info.pb
I20260619 14:32:30.337164 8897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:30.338400 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 8743
I20260619 14:32:30.341528 8897 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.2.62" port: 39059 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:32:30 dist-test-slave-70rm krb5kdc[8133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.2.60@KRBTEST.COM
Jun 19 14:32:30 dist-test-slave-70rm krb5kdc[8133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879550, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.2.60@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM
I20260619 14:32:30.352483 8217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:30.343401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37784 (local address 127.2.2.62:39059)
0619 14:32:30.343521 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:30.343525 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:30.343693 (+ 168us) server_negotiation.cc:407] Connection header received
0619 14:32:30.343816 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:30.343819 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:30.343863 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:30.343952 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:30.344582 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.345248 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.346353 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.346570 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.346680 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:30.349161 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:30.349168 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:30.349174 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:30.349204 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:30.350728 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:30.351162 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:30.351166 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:30.351168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:30.351252 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:30.351557 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:30.351561 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:30.351563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:30.351755 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:30.351863 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:30.352203 (+ 340us) server_negotiation.cc:299] Negotiation successful
0619 14:32:30.352321 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 14:32:30.353386 8900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:30.343198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:39059 (local address 127.0.0.1:37784)
0619 14:32:30.343563 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:30.343609 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:30.343726 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:30.344000 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:30.344004 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:30.344344 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:30.344459 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:30.344464 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.345365 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.345368 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:30.346237 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:30.346246 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.346667 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.346735 (+ 68us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:30.346830 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:30.347343 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:30.347358 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:30.349015 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:30.350840 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:30.350847 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:30.350856 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:30.351055 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:30.351345 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:30.351349 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:30.351351 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:30.351462 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:30.351858 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:30.351864 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:30.351996 (+ 132us) client_negotiation.cc:769] Sending connection context
0619 14:32:30.352168 (+ 172us) client_negotiation.cc:240] Negotiation successful
0619 14:32:30.352484 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":128,"threads_started":1}
I20260619 14:32:30.353396 8899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:30.339936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55948 (local address 127.2.2.60:45719)
0619 14:32:30.340408 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:30.340423 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:30.340460 (+ 37us) server_negotiation.cc:407] Connection header received
0619 14:32:30.341101 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:30.341119 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:30.341458 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:30.341812 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:30.342658 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.344017 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.344987 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.345584 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.345735 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:30.348731 (+ 2996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:30.348738 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:30.348752 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:30.348783 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:30.350519 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:30.351041 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:30.351047 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:30.351049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:30.351164 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:30.351443 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:30.351448 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:30.351450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:30.351832 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:30.352032 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:30.352220 (+ 188us) server_negotiation.cc:299] Negotiation successful
0619 14:32:30.352595 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":148,"threads_started":1}
I20260619 14:32:30.355494 8897 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.2.61" port: 36895 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:32:30 dist-test-slave-70rm krb5kdc[8133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879550, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.2.60@KRBTEST.COM for oryx/127.2.2.61@KRBTEST.COM
I20260619 14:32:30.364658 8900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:30.356037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.61:36895 (local address 127.0.0.1:37404)
0619 14:32:30.356137 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:30.356153 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:30.356235 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:30.356681 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:30.356683 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:30.356841 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:30.356945 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:30.356951 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.358061 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.358064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:30.358773 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:30.358783 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.359367 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.359386 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:30.359470 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:30.360043 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:30.360055 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:30.361414 (+ 1359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:30.363394 (+ 1980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:30.363396 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:30.363398 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:30.363608 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:30.363862 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:30.363864 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:30.363865 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:30.363937 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:30.364262 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:30.364264 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:30.364344 (+ 80us) client_negotiation.cc:769] Sending connection context
0619 14:32:30.364421 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 14:32:30.364531 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260619 14:32:30.364758 8901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:30.356151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37404 (local address 127.2.2.61:36895)
0619 14:32:30.356433 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:30.356443 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:30.356455 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:32:30.356491 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:30.356493 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:30.356533 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:30.356613 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:30.357097 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.357914 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.358919 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.359255 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.359350 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:30.361576 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:30.361580 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:30.361582 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:30.361607 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:30.363271 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:30.363704 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:30.363707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:30.363708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:30.363783 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:30.364012 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:30.364014 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:30.364015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:30.364189 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:30.364262 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:30.364475 (+ 213us) server_negotiation.cc:299] Negotiation successful
0619 14:32:30.364575 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":139,"threads_started":1}
I20260619 14:32:30.365518 8897 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.2.60" port: 45719 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:32:30 dist-test-slave-70rm krb5kdc[8133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879550, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.2.60@KRBTEST.COM for oryx/127.2.2.60@KRBTEST.COM
I20260619 14:32:30.373435 8900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:30.365954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.60:45719 (local address 127.0.0.1:55960)
0619 14:32:30.366101 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:30.366115 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:30.366282 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:30.366494 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:30.366497 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:30.366669 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:30.366732 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:30.366737 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.367813 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.367816 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:30.368326 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:30.368333 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.368752 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.368768 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:30.368840 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:30.369297 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:30.369311 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:30.370616 (+ 1305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:30.372267 (+ 1651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:30.372269 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:30.372270 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:30.372440 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:30.372661 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:30.372663 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:30.372664 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:30.372739 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:30.373043 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:30.373045 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:30.373115 (+ 70us) client_negotiation.cc:769] Sending connection context
0619 14:32:30.373190 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 14:32:30.373298 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 14:32:30.373492 8899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:30.366034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55960 (local address 127.2.2.60:45719)
0619 14:32:30.366147 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:30.366152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:30.366254 (+ 102us) server_negotiation.cc:407] Connection header received
0619 14:32:30.366363 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:30.366366 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:30.366408 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:30.366485 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:30.366835 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.367701 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.368433 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.368663 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.368769 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:30.370736 (+ 1967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:30.370739 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:30.370741 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:30.370773 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:30.372180 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:30.372509 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:30.372512 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:30.372513 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:30.372575 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:30.372813 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:30.372815 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:30.372816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:30.372970 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:30.373042 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:30.373221 (+ 179us) server_negotiation.cc:299] Negotiation successful
0619 14:32:30.373351 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":40}
I20260619 14:32:30.376559 8897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093: Bootstrap starting.
I20260619 14:32:30.379132 8897 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:30.379885 8897 log.cc:824] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093: Log is configured to *not* fsync() on all Append() calls
I20260619 14:32:30.381832 8897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093: No bootstrap required, opened a new log
I20260619 14:32:30.385066 8897 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf27644a9d194a898a1d93d6821e8b38" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 39059 } } peers { permanent_uuid: "e36baac4eca2403197084187e1360c08" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 36895 } } peers { permanent_uuid: "03d127796b8d4c3286344b40cff82093" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 45719 } }
I20260619 14:32:30.385313 8897 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:30.385399 8897 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03d127796b8d4c3286344b40cff82093, State: Initialized, Role: FOLLOWER
I20260619 14:32:30.385854 8897 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf27644a9d194a898a1d93d6821e8b38" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 39059 } } peers { permanent_uuid: "e36baac4eca2403197084187e1360c08" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 36895 } } peers { permanent_uuid: "03d127796b8d4c3286344b40cff82093" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 45719 } }
I20260619 14:32:30.386364 8903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf27644a9d194a898a1d93d6821e8b38" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 39059 } } peers { permanent_uuid: "e36baac4eca2403197084187e1360c08" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 36895 } } peers { permanent_uuid: "03d127796b8d4c3286344b40cff82093" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 45719 } } }
I20260619 14:32:30.386479 8903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:32:30.387126 8897 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 14:32:30.391623 8914 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:32:30.391719 8914 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:32:30.392009 8903 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:32:30.392168 8903 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf27644a9d194a898a1d93d6821e8b38" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 39059 } } peers { permanent_uuid: "e36baac4eca2403197084187e1360c08" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 36895 } } peers { permanent_uuid: "03d127796b8d4c3286344b40cff82093" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 45719 } }
I20260619 14:32:30.393092 8903 leader_election.cc:288] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf27644a9d194a898a1d93d6821e8b38 (127.2.2.62:39059), e36baac4eca2403197084187e1360c08 (127.2.2.61:36895)
I20260619 14:32:30.393474 8789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "03d127796b8d4c3286344b40cff82093" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e36baac4eca2403197084187e1360c08" is_pre_election: true
I20260619 14:32:30.393561 8188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "03d127796b8d4c3286344b40cff82093" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf27644a9d194a898a1d93d6821e8b38" is_pre_election: true
I20260619 14:32:30.393658 8789 raft_consensus.cc:1799] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [term 1 LEARNER]: Handling vote request from an unknown peer 03d127796b8d4c3286344b40cff82093
I20260619 14:32:30.393790 8789 raft_consensus.cc:2396] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate 03d127796b8d4c3286344b40cff82093 for term 1 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260619 14:32:30.394920 8846 leader_election.cc:302] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03d127796b8d4c3286344b40cff82093; no voters: cf27644a9d194a898a1d93d6821e8b38, e36baac4eca2403197084187e1360c08
I20260619 14:32:30.395107 8743 master_runner.cc:428] Detected that this master 03d127796b8d4c3286344b40cff82093 is joining an existing cluster
I20260619 14:32:30.395164 8743 master_runner.cc:432] Initiating AddMaster RPC to add 127.2.2.60:45719
I20260619 14:32:30.395354 8903 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:30.396709 8903 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260619 14:32:30.405099 8217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:30.397703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37792 (local address 127.2.2.62:39059)
0619 14:32:30.397831 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:30.397835 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:30.397845 (+ 10us) server_negotiation.cc:407] Connection header received
0619 14:32:30.397878 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:30.397880 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:30.397917 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:30.398004 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:30.398568 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.399120 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.399962 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.400184 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.400335 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:30.401410 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:30.401414 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:30.401417 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:30.401443 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:30.403108 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:30.403598 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:30.403602 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:30.403604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:30.403687 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:30.403952 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:30.403957 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:30.403959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:30.404134 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:30.404254 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:30.404801 (+ 547us) server_negotiation.cc:299] Negotiation successful
0619 14:32:30.404929 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 14:32:30.405635 8920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:30.397314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:39059 (local address 127.0.0.1:37792)
0619 14:32:30.397568 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:30.397584 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:30.397680 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:30.398022 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:30.398025 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:30.398291 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:30.398445 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:30.398453 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.399251 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.399254 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:30.399847 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:30.399854 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.400327 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.400342 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:30.400427 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:30.400953 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:30.400965 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:30.401277 (+ 312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:30.403232 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:30.403239 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:30.403242 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:30.403503 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:30.403778 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:30.403782 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:30.403784 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:30.403866 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:30.404481 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:30.404484 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:30.404605 (+ 121us) client_negotiation.cc:769] Sending connection context
0619 14:32:30.404739 (+ 134us) client_negotiation.cc:240] Negotiation successful
0619 14:32:30.405096 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"spinlock_wait_cycles":768,"thread_start_us":94,"threads_started":1}
I20260619 14:32:30.406980 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 8240
I20260619 14:32:30.407945 8901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:30.397467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37410 (local address 127.2.2.61:36895)
0619 14:32:30.397601 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:30.397605 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:30.398576 (+ 971us) server_negotiation.cc:407] Connection header received
0619 14:32:30.398772 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:30.398775 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:30.398826 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:30.398877 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:30.399583 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.400449 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.401631 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.401966 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.402078 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:30.403947 (+ 1869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:30.403951 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:30.403954 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:30.403984 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:30.405640 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:30.406199 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:30.406202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:30.406204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:30.406315 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:30.406607 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:30.406610 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:30.406612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:30.406817 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:30.406894 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:30.407673 (+ 779us) server_negotiation.cc:299] Negotiation successful
0619 14:32:30.407801 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 14:32:30.408442 8922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:30.397382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.61:36895 (local address 127.0.0.1:37410)
0619 14:32:30.398493 (+ 1111us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:30.398506 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:30.398648 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:30.399012 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:30.399015 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:30.399316 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:30.399445 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:30.399453 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.400587 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.400590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:30.401501 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:30.401511 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.402077 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.402094 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:30.402219 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:30.403245 (+ 1026us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:30.403274 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:30.403763 (+ 489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:30.405769 (+ 2006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:30.405772 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:30.405775 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:30.406047 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:30.406429 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:30.406432 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:30.406434 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:30.406514 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:30.406947 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:30.406950 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:30.407035 (+ 85us) client_negotiation.cc:769] Sending connection context
0619 14:32:30.408067 (+ 1032us) client_negotiation.cc:240] Negotiation successful
0619 14:32:30.408236 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1059,"spinlock_wait_cycles":4480,"thread_start_us":110,"threads_started":1}
I20260619 14:32:30.414628 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.1:45703
--local_ip_for_outbound_sockets=127.2.2.1
--tserver_master_addrs=127.2.2.62:39059
--webserver_port=45677
--webserver_interface=127.2.2.1
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/krb5kdc/oryx_127.2.2.1.keytab
--principal=oryx/127.2.2.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:45229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.2.62:39059,127.2.2.61:36895
--tserver_master_addrs=127.2.2.62:39059,127.2.2.61:36895,127.2.2.60:45719 with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:32:30 dist-test-slave-70rm krb5kdc[8133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879548, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.2.62@KRBTEST.COM for oryx/127.2.2.60@KRBTEST.COM
I20260619 14:32:30.438428 8735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:30.410497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.60:45719 (local address 127.0.0.1:55980)
0619 14:32:30.410601 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:30.410614 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:30.410690 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:30.414473 (+ 3783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:30.414476 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:30.414672 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:30.414815 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:30.414825 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.415810 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.415814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:30.416756 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:30.416769 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.417373 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.417392 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:30.417471 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:30.417984 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:30.417996 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:30.419490 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:30.436692 (+ 17202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:30.436696 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:30.436699 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:30.436983 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:30.437305 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:30.437308 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:30.437310 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:30.437396 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:30.437851 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:30.437855 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:30.437958 (+ 103us) client_negotiation.cc:769] Sending connection context
0619 14:32:30.438040 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 14:32:30.438247 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260619 14:32:30.439018 8899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:30.398238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55970 (local address 127.2.2.60:45719)
0619 14:32:30.398338 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:30.398342 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:30.408811 (+ 10469us) server_negotiation.cc:407] Connection header received
0619 14:32:30.409048 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:30.409051 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:30.409120 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:30.409190 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:30.409695 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.410458 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.411566 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.411893 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.411958 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:30.435696 (+ 23738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:30.435701 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:30.435704 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:30.435728 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:30.437414 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:30.437889 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:30.437892 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:30.437894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:30.437963 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:30.438232 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:30.438235 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:30.438237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:30.438415 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:30.438481 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:30.438762 (+ 281us) server_negotiation.cc:299] Negotiation successful
0619 14:32:30.438870 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 14:32:30.439496 8921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:30.398256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.60:45719 (local address 127.0.0.1:55970)
0619 14:32:30.408545 (+ 10289us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:30.408561 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:30.408899 (+ 338us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:30.409326 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:30.409329 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:30.409500 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:30.409586 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:30.409590 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.410569 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.410572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:30.411449 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:30.411459 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.412072 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.412087 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:30.412171 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:30.434839 (+ 22668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:30.434861 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:30.435472 (+ 611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:30.437545 (+ 2073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:30.437548 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:30.437549 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:30.437787 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:30.438052 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:30.438055 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:30.438057 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:30.438148 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:30.438534 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:30.438537 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:30.438609 (+ 72us) client_negotiation.cc:769] Sending connection context
0619 14:32:30.439200 (+ 591us) client_negotiation.cc:240] Negotiation successful
0619 14:32:30.439329 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10129,"thread_start_us":88,"threads_started":1}
I20260619 14:32:30.439818 8923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:30.410752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55980 (local address 127.2.2.60:45719)
0619 14:32:30.414225 (+ 3473us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:30.414231 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:30.414243 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:32:30.414303 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:30.414305 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:30.414346 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:30.414447 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:30.414953 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.415680 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.416892 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.417257 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.417361 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:30.419647 (+ 2286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:30.419651 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:30.419654 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:30.419687 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:30.436537 (+ 16850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:30.437109 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:30.437113 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:30.437115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:30.437194 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:30.437499 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:30.437502 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:30.437503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:30.437737 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:30.439003 (+ 1266us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:30.439313 (+ 310us) server_negotiation.cc:299] Negotiation successful
0619 14:32:30.439430 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3369,"thread_start_us":173,"threads_started":1}
I20260619 14:32:30.440860 8174 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.2.2.60:45719: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "03d127796b8d4c3286344b40cff82093"
member_type: NON_VOTER
last_known_addr {
host: "127.2.2.60"
port: 45719
}
attrs {
promote: true
}
}
dest_uuid: "cf27644a9d194a898a1d93d6821e8b38"
cas_config_opid_index: 5
I20260619 14:32:30.441535 8174 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf27644a9d194a898a1d93d6821e8b38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cf27644a9d194a898a1d93d6821e8b38" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 39059 } } peers { permanent_uuid: "e36baac4eca2403197084187e1360c08" member_type: NON_VOTER last_known_addr { host: "127.2.2.61" port: 36895 } attrs { promote: true } } peers { permanent_uuid: "03d127796b8d4c3286344b40cff82093" member_type: NON_VOTER last_known_addr { host: "127.2.2.60" port: 45719 } attrs { promote: true } }
I20260619 14:32:30.442760 8738 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cf27644a9d194a898a1d93d6821e8b38 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 03d127796b8d4c3286344b40cff82093 (127.2.2.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cf27644a9d194a898a1d93d6821e8b38" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 39059 } } peers { permanent_uuid: "e36baac4eca2403197084187e1360c08" member_type: NON_VOTER last_known_addr { host: "127.2.2.61" port: 36895 } attrs { promote: true } } peers { permanent_uuid: "03d127796b8d4c3286344b40cff82093" member_type: NON_VOTER last_known_addr { host: "127.2.2.60" port: 45719 } attrs { promote: true } } }
I20260619 14:32:30.443522 8871 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093 [term 1 FOLLOWER]: Refusing update from remote peer cf27644a9d194a898a1d93d6821e8b38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 14:32:30.444290 8738 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.2.2.60:45719
I20260619 14:32:30.444453 8738 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf27644a9d194a898a1d93d6821e8b38 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "cf27644a9d194a898a1d93d6821e8b38" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cf27644a9d194a898a1d93d6821e8b38" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 39059 } } peers { permanent_uuid: "e36baac4eca2403197084187e1360c08" member_type: NON_VOTER last_known_addr { host: "127.2.2.61" port: 36895 } attrs { promote: true } } peers { permanent_uuid: "03d127796b8d4c3286344b40cff82093" member_type: NON_VOTER last_known_addr { host: "127.2.2.60" port: 45719 } attrs { promote: true } } }
I20260619 14:32:30.444536 8738 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf27644a9d194a898a1d93d6821e8b38 [sys.catalog]: This master's current role is: LEADER
I20260619 14:32:30.444923 8743 master.cc:537] Master@127.2.2.60:45719 shutting down...
I20260619 14:32:30.446343 8739 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cf27644a9d194a898a1d93d6821e8b38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03d127796b8d4c3286344b40cff82093" member_type: NON_VOTER last_known_addr { host: "127.2.2.60" port: 45719 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:32:30.447686 8735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:30.442690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.61:36895 (local address 127.0.0.1:37424)
0619 14:32:30.443148 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:30.443162 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:30.443242 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:30.443630 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:30.443633 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:30.443648 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:32:30.443756 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:30.443762 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.444804 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.444808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:30.446078 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:30.446089 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.447244 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.447248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:30.447345 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:30.447395 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 14:32:30.447467 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 14:32:30.447546 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387}
I20260619 14:32:30.448135 8901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:30.442777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37424 (local address 127.2.2.61:36895)
0619 14:32:30.442891 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:30.442895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:30.443322 (+ 427us) server_negotiation.cc:407] Connection header received
0619 14:32:30.443375 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:30.443378 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:30.443436 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:30.443515 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:32:30.443881 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.444659 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.446252 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.446930 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.447032 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:30.447182 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:30.447933 (+ 751us) server_negotiation.cc:299] Negotiation successful
0619 14:32:30.448017 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 14:32:30.448563 8789 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [term 1 LEARNER]: Refusing update from remote peer cf27644a9d194a898a1d93d6821e8b38: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 14:32:30.449016 8739 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cf27644a9d194a898a1d93d6821e8b38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e36baac4eca2403197084187e1360c08" member_type: NON_VOTER last_known_addr { host: "127.2.2.61" port: 36895 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260619 14:32:30.449183 8165 consensus_peers.cc:487] T 00000000000000000000000000000000 P cf27644a9d194a898a1d93d6821e8b38 -> Peer 03d127796b8d4c3286344b40cff82093 (127.2.2.60:45719): Couldn't send request to peer 03d127796b8d4c3286344b40cff82093. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 14:32:30.450932 8789 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 03d127796b8d4c3286344b40cff82093 (127.2.2.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cf27644a9d194a898a1d93d6821e8b38" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 39059 } } peers { permanent_uuid: "e36baac4eca2403197084187e1360c08" member_type: NON_VOTER last_known_addr { host: "127.2.2.61" port: 36895 } attrs { promote: true } } peers { permanent_uuid: "03d127796b8d4c3286344b40cff82093" member_type: NON_VOTER last_known_addr { host: "127.2.2.60" port: 45719 } attrs { promote: true } } }
I20260619 14:32:30.452183 8743 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:32:30.452477 8743 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:32:30.452576 8743 tablet_replica.cc:330] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093: stopping tablet replica
I20260619 14:32:30.453423 8818 sys_catalog.cc:455] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf27644a9d194a898a1d93d6821e8b38. Latest consensus state: current_term: 1 leader_uuid: "cf27644a9d194a898a1d93d6821e8b38" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cf27644a9d194a898a1d93d6821e8b38" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 39059 } } peers { permanent_uuid: "e36baac4eca2403197084187e1360c08" member_type: NON_VOTER last_known_addr { host: "127.2.2.61" port: 36895 } attrs { promote: true } } peers { permanent_uuid: "03d127796b8d4c3286344b40cff82093" member_type: NON_VOTER last_known_addr { host: "127.2.2.60" port: 45719 } attrs { promote: true } } }
I20260619 14:32:30.453544 8818 sys_catalog.cc:458] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [sys.catalog]: This master's current role is: LEARNER
I20260619 14:32:30.453822 8926 sys_catalog.cc:455] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cf27644a9d194a898a1d93d6821e8b38" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cf27644a9d194a898a1d93d6821e8b38" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 39059 } } peers { permanent_uuid: "e36baac4eca2403197084187e1360c08" member_type: NON_VOTER last_known_addr { host: "127.2.2.61" port: 36895 } attrs { promote: true } } peers { permanent_uuid: "03d127796b8d4c3286344b40cff82093" member_type: NON_VOTER last_known_addr { host: "127.2.2.60" port: 45719 } attrs { promote: true } } }
I20260619 14:32:30.453969 8926 sys_catalog.cc:458] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [sys.catalog]: This master's current role is: LEARNER
I20260619 14:32:30.457835 8743 master.cc:552] Master@127.2.2.60:45719 shutdown complete.
I20260619 14:32:30.457945 8743 master_runner.cc:305] Clearing existing system tablet
I20260619 14:32:30.458321 8743 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:30.458418 8743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:30.459661 8739 raft_consensus.cc:1050] T 00000000000000000000000000000000 P cf27644a9d194a898a1d93d6821e8b38: attempting to promote NON_VOTER e36baac4eca2403197084187e1360c08 to VOTER
I20260619 14:32:30.460207 8743 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 14:32:30.460155 8739 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf27644a9d194a898a1d93d6821e8b38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cf27644a9d194a898a1d93d6821e8b38" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 39059 } } peers { permanent_uuid: "e36baac4eca2403197084187e1360c08" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 36895 } attrs { promote: false } } peers { permanent_uuid: "03d127796b8d4c3286344b40cff82093" member_type: NON_VOTER last_known_addr { host: "127.2.2.60" port: 45719 } attrs { promote: true } }
I20260619 14:32:30.461122 8929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:30.461334 8743 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:30.461438 8743 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 14:32:30.461468 8743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-2/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-2/wal
uuid: "03d127796b8d4c3286344b40cff82093"
format_stamp: "Formatted at 2026-06-19 14:32:30 on dist-test-slave-70rm"
server_key: "04e697cab2bdc8560576834a6066f799"
server_key_iv: "d29990d4c698c7f5563489ab0a59160d"
server_key_version: "encryptionkey@0"
I20260619 14:32:30.461944 8735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:30.461239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.60:45719 (local address 127.0.0.1:55994)
0619 14:32:30.461791 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:30.461863 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.60:45719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457}
I20260619 14:32:30.462440 8789 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [term 1 LEARNER]: Refusing update from remote peer cf27644a9d194a898a1d93d6821e8b38: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260619 14:32:30.463524 8735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:30.463191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.60:45719 (local address 127.0.0.1:56008)
0619 14:32:30.463401 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:30.463454 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.60:45719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
I20260619 14:32:30.463507 8743 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:32:30.463632 8738 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cf27644a9d194a898a1d93d6821e8b38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e36baac4eca2403197084187e1360c08" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 36895 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260619 14:32:30.464684 8165 consensus_peers.cc:487] T 00000000000000000000000000000000 P cf27644a9d194a898a1d93d6821e8b38 -> Peer 03d127796b8d4c3286344b40cff82093 (127.2.2.60:45719): Couldn't send request to peer 03d127796b8d4c3286344b40cff82093. Status: Network error: Client connection negotiation failed: client connection to 127.2.2.60:45719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 14:32:30.465597 8739 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cf27644a9d194a898a1d93d6821e8b38 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e36baac4eca2403197084187e1360c08 (127.2.2.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cf27644a9d194a898a1d93d6821e8b38" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 39059 } } peers { permanent_uuid: "e36baac4eca2403197084187e1360c08" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 36895 } attrs { promote: false } } peers { permanent_uuid: "03d127796b8d4c3286344b40cff82093" member_type: NON_VOTER last_known_addr { host: "127.2.2.60" port: 45719 } attrs { promote: true } } }
I20260619 14:32:30.466420 8743 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 14:32:30.466565 8743 log.cc:1196] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093: Deleting WAL directory at /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260619 14:32:30.466969 8743 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093: Deleting consensus metadata
I20260619 14:32:30.467201 8743 master_runner.cc:315] Copying system tablet from 127.2.2.62:39059
I20260619 14:32:30.467579 8789 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e36baac4eca2403197084187e1360c08 (127.2.2.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cf27644a9d194a898a1d93d6821e8b38" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 39059 } } peers { permanent_uuid: "e36baac4eca2403197084187e1360c08" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 36895 } attrs { promote: false } } peers { permanent_uuid: "03d127796b8d4c3286344b40cff82093" member_type: NON_VOTER last_known_addr { host: "127.2.2.60" port: 45719 } attrs { promote: true } } }
I20260619 14:32:30.468482 8743 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093: tablet copy: Beginning tablet copy session from remote peer at address 127.2.2.62:39059
I20260619 14:32:30.469163 8926 sys_catalog.cc:455] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cf27644a9d194a898a1d93d6821e8b38" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cf27644a9d194a898a1d93d6821e8b38" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 39059 } } peers { permanent_uuid: "e36baac4eca2403197084187e1360c08" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 36895 } attrs { promote: false } } peers { permanent_uuid: "03d127796b8d4c3286344b40cff82093" member_type: NON_VOTER last_known_addr { host: "127.2.2.60" port: 45719 } attrs { promote: true } } }
I20260619 14:32:30.469292 8926 sys_catalog.cc:458] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:32:30.469825 8739 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf27644a9d194a898a1d93d6821e8b38 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "cf27644a9d194a898a1d93d6821e8b38" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cf27644a9d194a898a1d93d6821e8b38" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 39059 } } peers { permanent_uuid: "e36baac4eca2403197084187e1360c08" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 36895 } attrs { promote: false } } peers { permanent_uuid: "03d127796b8d4c3286344b40cff82093" member_type: NON_VOTER last_known_addr { host: "127.2.2.60" port: 45719 } attrs { promote: true } } }
I20260619 14:32:30.469933 8739 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf27644a9d194a898a1d93d6821e8b38 [sys.catalog]: This master's current role is: LEADER
I20260619 14:32:30.470108 8738 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf27644a9d194a898a1d93d6821e8b38 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cf27644a9d194a898a1d93d6821e8b38" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cf27644a9d194a898a1d93d6821e8b38" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 39059 } } peers { permanent_uuid: "e36baac4eca2403197084187e1360c08" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 36895 } attrs { promote: false } } peers { permanent_uuid: "03d127796b8d4c3286344b40cff82093" member_type: NON_VOTER last_known_addr { host: "127.2.2.60" port: 45719 } attrs { promote: true } } }
I20260619 14:32:30.470254 8738 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf27644a9d194a898a1d93d6821e8b38 [sys.catalog]: This master's current role is: LEADER
I20260619 14:32:30.478458 8217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:30.469258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37800 (local address 127.2.2.62:39059)
0619 14:32:30.469368 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:30.469373 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:30.469913 (+ 540us) server_negotiation.cc:407] Connection header received
0619 14:32:30.470201 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:30.470205 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:30.470253 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:30.470304 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:30.470994 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.471796 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.472986 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.473321 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.473452 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:30.474804 (+ 1352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:30.474810 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:30.474815 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:30.474848 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:30.476505 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:30.476989 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:30.476992 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:30.476994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:30.477080 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:30.477378 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:30.477381 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:30.477383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:30.477615 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:30.477727 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:30.478098 (+ 371us) server_negotiation.cc:299] Negotiation successful
0619 14:32:30.478304 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 14:32:30.478468 8935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:30.469541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:39059 (local address 127.0.0.1:37800)
0619 14:32:30.469836 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:30.469849 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:30.470061 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:30.470501 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:30.470504 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:30.470732 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:30.470862 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:30.470870 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.471948 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.471952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:30.472849 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:30.472860 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.473439 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.473458 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:30.473553 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:30.474150 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:30.474166 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:30.474632 (+ 466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:30.476625 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:30.476628 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:30.476630 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:30.476889 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:30.477175 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:30.477178 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:30.477180 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:30.477292 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:30.477742 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:30.477745 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:30.477913 (+ 168us) client_negotiation.cc:769] Sending connection context
0619 14:32:30.478028 (+ 115us) client_negotiation.cc:240] Negotiation successful
0619 14:32:30.478242 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":63,"threads_started":1}
I20260619 14:32:30.479306 8198 tablet_copy_service.cc:140] P cf27644a9d194a898a1d93d6821e8b38: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 03d127796b8d4c3286344b40cff82093 ({username='oryx', principal='oryx/127.2.2.60@KRBTEST.COM'} at 127.0.0.1:37800)
I20260619 14:32:30.479408 8198 tablet_copy_service.cc:161] P cf27644a9d194a898a1d93d6821e8b38: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 03d127796b8d4c3286344b40cff82093 at {username='oryx', principal='oryx/127.2.2.60@KRBTEST.COM'} at 127.0.0.1:37800: session id = 03d127796b8d4c3286344b40cff82093-00000000000000000000000000000000
I20260619 14:32:30.480464 8198 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P cf27644a9d194a898a1d93d6821e8b38: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 14:32:30.481858 8743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 14:32:30.484277 8743 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093: tablet copy: Starting download of 0 data blocks...
I20260619 14:32:30.484465 8743 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093: tablet copy: Starting download of 1 WAL segments...
I20260619 14:32:30.485963 8743 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 14:32:30.487802 8198 tablet_copy_service.cc:342] P cf27644a9d194a898a1d93d6821e8b38: Request end of tablet copy session 03d127796b8d4c3286344b40cff82093-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.2.2.60@KRBTEST.COM'} at 127.0.0.1:37800
I20260619 14:32:30.487888 8198 tablet_copy_service.cc:433] P cf27644a9d194a898a1d93d6821e8b38: ending tablet copy session 03d127796b8d4c3286344b40cff82093-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 03d127796b8d4c3286344b40cff82093
I20260619 14:32:30.495067 8743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:32:30.496861 8937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:30.496948 8938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:30.497217 8940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:30.497902 8743 server_base.cc:1034] running on GCE node
I20260619 14:32:30.498065 8743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:30.498378 8743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:30.499498 8743 hybrid_clock.cc:648] HybridClock initialized: now 1781879550499485 us; error 31 us; skew 500 ppm
Jun 19 14:32:30 dist-test-slave-70rm krb5kdc[8133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879550, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.2.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:32:30.508502 8743 init.cc:377] Logged in from keytab as oryx/127.2.2.60@KRBTEST.COM (short username oryx)
I20260619 14:32:30.509014 8743 webserver.cc:460] Webserver started at http://127.2.2.60:39407/ using document root <none> and password file <none>
I20260619 14:32:30.509235 8743 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:30.509289 8743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:30.514644 8743 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.000s sys 0.001s
I20260619 14:32:30.515758 8947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:30.516033 8743 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 14:32:30.516149 8743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-2/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-2/wal
uuid: "03d127796b8d4c3286344b40cff82093"
format_stamp: "Formatted at 2026-06-19 14:32:30 on dist-test-slave-70rm"
server_key: "04e697cab2bdc8560576834a6066f799"
server_key_iv: "d29990d4c698c7f5563489ab0a59160d"
server_key_version: "encryptionkey@0"
I20260619 14:32:30.516299 8743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:30.523610 8503 heartbeater.cc:503] Master 127.2.2.62:39059 was elected leader, sending a full tablet report...
I20260619 14:32:30.531282 8743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:30.533805 8743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:30.539045 8743 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.60:45719
I20260619 14:32:30.539155 8999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.60:45719 every 8 connection(s)
I20260619 14:32:30.539876 8743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/master-2/data/info.pb
I20260619 14:32:30.541864 9000 sys_catalog.cc:263] Verifying existing consensus state
I20260619 14:32:30.543041 9000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093: Bootstrap starting.
I20260619 14:32:30.554709 9000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:32:30.555056 9000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093: Bootstrap complete.
I20260619 14:32:30.555686 9000 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cf27644a9d194a898a1d93d6821e8b38" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 39059 } } peers { permanent_uuid: "e36baac4eca2403197084187e1360c08" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 36895 } attrs { promote: false } } peers { permanent_uuid: "03d127796b8d4c3286344b40cff82093" member_type: NON_VOTER last_known_addr { host: "127.2.2.60" port: 45719 } attrs { promote: true } }
I20260619 14:32:30.555838 9000 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 03d127796b8d4c3286344b40cff82093, State: Initialized, Role: LEARNER
I20260619 14:32:30.555941 9000 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cf27644a9d194a898a1d93d6821e8b38" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 39059 } } peers { permanent_uuid: "e36baac4eca2403197084187e1360c08" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 36895 } attrs { promote: false } } peers { permanent_uuid: "03d127796b8d4c3286344b40cff82093" member_type: NON_VOTER last_known_addr { host: "127.2.2.60" port: 45719 } attrs { promote: true } }
I20260619 14:32:30.556444 9000 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:32:30.556324 9003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cf27644a9d194a898a1d93d6821e8b38" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 39059 } } peers { permanent_uuid: "e36baac4eca2403197084187e1360c08" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 36895 } attrs { promote: false } } peers { permanent_uuid: "03d127796b8d4c3286344b40cff82093" member_type: NON_VOTER last_known_addr { host: "127.2.2.60" port: 45719 } attrs { promote: true } } }
I20260619 14:32:30.556653 9003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093 [sys.catalog]: This master's current role is: LEARNER
I20260619 14:32:30.561443 9014 catalog_manager.cc:1259] Loaded cluster ID: 96382bd00e6e4b0792b1bfa035e796dc
I20260619 14:32:30.561524 9014 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093: loading cluster ID for follower catalog manager: success
I20260619 14:32:30.563764 9014 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093: acquiring CA information for follower catalog manager: success
I20260619 14:32:30.565222 9014 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 14:32:30.568629 8924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:30.568938 8924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:30.569043 8924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:30.573307 8924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:30.573419 8924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:30.573520 8924 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.1
I20260619 14:32:30.579413 8924 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:45229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/krb5kdc/oryx_127.2.2.1.keytab
--principal=oryx/127.2.2.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.1:45703
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.2.1
--webserver_port=45677
--tserver_master_addrs=127.2.2.62:39059,127.2.2.61:36895,127.2.2.60:45719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8924
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.1
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:32:30.580595 8924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:30.581629 8924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:32:30.591964 8924 server_base.cc:1034] running on GCE node
W20260619 14:32:30.591910 9022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:30.591913 9020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:30.591913 9019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:30.592639 8924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:30.593372 8924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:30.594595 8924 hybrid_clock.cc:648] HybridClock initialized: now 1781879550594566 us; error 50 us; skew 500 ppm
Jun 19 14:32:30 dist-test-slave-70rm krb5kdc[8133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879550, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.2.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:32:30.597465 8924 init.cc:377] Logged in from keytab as oryx/127.2.2.1@KRBTEST.COM (short username oryx)
I20260619 14:32:30.598488 8924 webserver.cc:460] Webserver started at http://127.2.2.1:45677/ using document root <none> and password file <none>
I20260619 14:32:30.599092 8924 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:30.599215 8924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:30.603196 8924 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:32:30.605638 9029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:30.607081 8924 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:32:30.607229 8924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-0/wal
uuid: "c81bd9d0e5f44e0fa56ac4dcc2401a27"
format_stamp: "Formatted at 2026-06-19 14:32:29 on dist-test-slave-70rm"
server_key: "60f5eaed1e95d8c80339ac3dfbedf157"
server_key_iv: "d4810e94317241558a02157c1a570b77"
server_key_version: "encryptionkey@0"
I20260619 14:32:30.607806 8924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:30.619288 8924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:30.622797 8924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:32:30.623005 8924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:30.623601 8924 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:32:30.624541 8924 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:32:30.624610 8924 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:30.624681 8924 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:32:30.624734 8924 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:30.634198 8924 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.1:45703
I20260619 14:32:30.634254 9142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.1:45703 every 8 connection(s)
I20260619 14:32:30.635349 8924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
I20260619 14:32:30.644731 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 8924
I20260619 14:32:30.644943 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 8374
Jun 19 14:32:30 dist-test-slave-70rm krb5kdc[8133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879550, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.2.1@KRBTEST.COM for oryx/127.2.2.60@KRBTEST.COM
Jun 19 14:32:30 dist-test-slave-70rm krb5kdc[8133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879550, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.2.1@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM
Jun 19 14:32:30 dist-test-slave-70rm krb5kdc[8133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879550, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.2.1@KRBTEST.COM for oryx/127.2.2.61@KRBTEST.COM
I20260619 14:32:30.652696 9147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:30.637299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:41513 (local address 127.2.2.60:45719)
0619 14:32:30.637886 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:30.637891 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:30.638871 (+ 980us) server_negotiation.cc:407] Connection header received
0619 14:32:30.639678 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:30.639680 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:30.639727 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:30.639840 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:30.641288 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.641994 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.643909 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.644287 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.644358 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:30.648169 (+ 3811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:30.648173 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:30.648176 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:30.648212 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:30.649759 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:30.650584 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:30.650590 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:30.650596 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:30.650696 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:30.651184 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:30.651191 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:30.651195 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:30.651459 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:30.651546 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:30.652323 (+ 777us) server_negotiation.cc:299] Negotiation successful
0619 14:32:30.652478 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":95,"threads_started":1}
I20260619 14:32:30.653457 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.2:34039
--local_ip_for_outbound_sockets=127.2.2.2
--tserver_master_addrs=127.2.2.62:39059
--webserver_port=34185
--webserver_interface=127.2.2.2
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/krb5kdc/oryx_127.2.2.2.keytab
--principal=oryx/127.2.2.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:45229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.2.62:39059,127.2.2.61:36895
--tserver_master_addrs=127.2.2.62:39059,127.2.2.61:36895,127.2.2.60:45719 with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:32:30.654182 8217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:30.637368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:48813 (local address 127.2.2.62:39059)
0619 14:32:30.637497 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:30.637501 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:30.638430 (+ 929us) server_negotiation.cc:407] Connection header received
0619 14:32:30.640001 (+ 1571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:30.640005 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:30.640063 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:30.640150 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:30.641303 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.642112 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.643910 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.644259 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.644345 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:30.650048 (+ 5703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:30.650054 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:30.650058 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:30.650091 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:30.652264 (+ 2173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:30.652819 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:30.652823 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:30.652825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:30.652947 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:30.653246 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:30.653249 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:30.653251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:30.653454 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:30.653527 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:30.653890 (+ 363us) server_negotiation.cc:299] Negotiation successful
0619 14:32:30.654001 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 14:32:30.654493 8901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:30.637579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:44461 (local address 127.2.2.61:36895)
0619 14:32:30.637725 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:30.637731 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:30.638632 (+ 901us) server_negotiation.cc:407] Connection header received
0619 14:32:30.639856 (+ 1224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:30.639860 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:30.639920 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:30.639999 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:30.641303 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.642101 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.643913 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.644292 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.644377 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:30.650549 (+ 6172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:30.650556 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:30.650562 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:30.650598 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:30.652345 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:30.652972 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:30.652979 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:30.652984 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:30.653073 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:30.653398 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:30.653402 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:30.653403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:30.653635 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:30.653710 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:30.654075 (+ 365us) server_negotiation.cc:299] Negotiation successful
0619 14:32:30.654254 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 14:32:30.655347 9150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:30.637734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:39059 (local address 127.2.2.1:48813)
0619 14:32:30.638274 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:30.638326 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:30.639867 (+ 1541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:30.640182 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:30.640185 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:30.640598 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:30.641130 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:30.641150 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.642242 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.642245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:30.643762 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:30.643769 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.644512 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.644567 (+ 55us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:30.644971 (+ 404us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:30.645864 (+ 893us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:30.645876 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:30.649822 (+ 3946us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:30.652378 (+ 2556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:30.652381 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:30.652383 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:30.652697 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:30.653062 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:30.653064 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:30.653066 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:30.653151 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:30.653621 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:30.653625 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:30.653718 (+ 93us) client_negotiation.cc:769] Sending connection context
0619 14:32:30.654389 (+ 671us) client_negotiation.cc:240] Negotiation successful
0619 14:32:30.654531 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":522,"spinlock_wait_cycles":19072,"thread_start_us":169,"threads_started":1}
I20260619 14:32:30.655359 9149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:30.637709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.60:45719 (local address 127.2.2.1:41513)
0619 14:32:30.638782 (+ 1073us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:30.638792 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:30.639506 (+ 714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:30.640018 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:30.640027 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:30.640567 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:30.641137 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:30.641148 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.642178 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.642184 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:30.643759 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:30.643769 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.644504 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.644567 (+ 63us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:30.644971 (+ 404us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:30.645795 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:30.645827 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:30.647935 (+ 2108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:30.649884 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:30.649890 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:30.649908 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:30.650456 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:30.650842 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:30.650848 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:30.650852 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:30.651059 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:30.651623 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:30.651630 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:30.652000 (+ 370us) client_negotiation.cc:769] Sending connection context
0619 14:32:30.653112 (+ 1112us) client_negotiation.cc:240] Negotiation successful
0619 14:32:30.653510 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":990,"mutex_wait_us":468,"spinlock_wait_cycles":1280,"thread_start_us":101,"threads_started":1}
I20260619 14:32:30.655627 9148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:30.637710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.61:36895 (local address 127.2.2.1:44461)
0619 14:32:30.638560 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:30.638571 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:30.639748 (+ 1177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:30.640020 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:30.640027 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:30.640605 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:30.641130 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:30.641150 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.642281 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.642286 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:30.643748 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:30.643763 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.644544 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.644577 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:30.644943 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:30.648210 (+ 3267us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:30.648242 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:30.650381 (+ 2139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:30.652479 (+ 2098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:30.652485 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:30.652490 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:30.652856 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:30.653191 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:30.653196 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:30.653198 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:30.653296 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:30.653785 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:30.653790 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:30.653896 (+ 106us) client_negotiation.cc:769] Sending connection context
0619 14:32:30.654698 (+ 802us) client_negotiation.cc:240] Negotiation successful
0619 14:32:30.654880 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":766,"mutex_wait_us":512,"spinlock_wait_cycles":3456,"thread_start_us":119,"threads_started":1}
I20260619 14:32:30.656816 9145 heartbeater.cc:347] Connected to a master server at 127.2.2.61:36895
I20260619 14:32:30.656941 9144 heartbeater.cc:347] Connected to a master server at 127.2.2.60:45719
I20260619 14:32:30.657078 9145 heartbeater.cc:464] Registering TS with master...
I20260619 14:32:30.657083 9144 heartbeater.cc:464] Registering TS with master...
I20260619 14:32:30.656852 9143 heartbeater.cc:347] Connected to a master server at 127.2.2.62:39059
I20260619 14:32:30.657667 9144 heartbeater.cc:511] Master 127.2.2.60:45719 requested a full tablet report, sending...
I20260619 14:32:30.657783 9143 heartbeater.cc:464] Registering TS with master...
I20260619 14:32:30.657670 9145 heartbeater.cc:511] Master 127.2.2.61:36895 requested a full tablet report, sending...
I20260619 14:32:30.657987 9143 heartbeater.cc:511] Master 127.2.2.62:39059 requested a full tablet report, sending...
I20260619 14:32:30.659147 8174 ts_manager.cc:194] Re-registered known tserver with Master: c81bd9d0e5f44e0fa56ac4dcc2401a27 (127.2.2.1:45703)
I20260619 14:32:30.659345 8964 ts_manager.cc:194] Registered new tserver with Master: c81bd9d0e5f44e0fa56ac4dcc2401a27 (127.2.2.1:45703)
I20260619 14:32:30.659353 8776 ts_manager.cc:194] Registered new tserver with Master: c81bd9d0e5f44e0fa56ac4dcc2401a27 (127.2.2.1:45703)
I20260619 14:32:30.659875 8174 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.2.1@KRBTEST.COM'} at 127.2.2.1:48813
I20260619 14:32:30.762032 8637 heartbeater.cc:503] Master 127.2.2.62:39059 was elected leader, sending a full tablet report...
W20260619 14:32:30.776295 9151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:30.776621 9151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:30.776697 9151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:30.780777 9151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:30.780875 9151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:30.780970 9151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.2
I20260619 14:32:30.786932 9151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:45229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/krb5kdc/oryx_127.2.2.2.keytab
--principal=oryx/127.2.2.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.2:34039
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.2.2
--webserver_port=34185
--tserver_master_addrs=127.2.2.62:39059,127.2.2.61:36895,127.2.2.60:45719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.2
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:32:30.788086 9151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:30.789145 9151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:32:30.799193 9159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:30.799193 9156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:30.799517 9151 server_base.cc:1034] running on GCE node
W20260619 14:32:30.799245 9157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:30.799933 9151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:30.800680 9151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:30.801875 9151 hybrid_clock.cc:648] HybridClock initialized: now 1781879550801856 us; error 34 us; skew 500 ppm
Jun 19 14:32:30 dist-test-slave-70rm krb5kdc[8133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879550, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.2.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:32:30.804813 9151 init.cc:377] Logged in from keytab as oryx/127.2.2.2@KRBTEST.COM (short username oryx)
I20260619 14:32:30.805876 9151 webserver.cc:460] Webserver started at http://127.2.2.2:34185/ using document root <none> and password file <none>
I20260619 14:32:30.806485 9151 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:30.806562 9151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:30.810491 9151 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:32:30.812791 9166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:30.814323 9151 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:32:30.814450 9151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-1/wal
uuid: "3cc7f187edfc4edabbb95a0f1bdce56a"
format_stamp: "Formatted at 2026-06-19 14:32:29 on dist-test-slave-70rm"
server_key: "ac7890c3135cb9d16e1355506936326d"
server_key_iv: "772972fdb895fd2cb0e018aec1422fcd"
server_key_version: "encryptionkey@0"
I20260619 14:32:30.815018 9151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:30.844017 9151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:30.847275 9151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:32:30.847472 9151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:30.848094 9151 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:32:30.848985 9151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:32:30.849040 9151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:30.849094 9151 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:32:30.849123 9151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:30.853816 8735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:30.849749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.60:45719 (local address 127.0.0.1:56016)
0619 14:32:30.849884 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:30.849923 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:30.850055 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:30.850453 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:30.850457 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:30.850476 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:32:30.850576 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:30.850582 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.851325 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.851329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:30.852516 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:30.852531 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.853391 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.853394 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:30.853480 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:30.853503 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 14:32:30.853591 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 14:32:30.853678 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260619 14:32:30.853816 9147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:30.850053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56016 (local address 127.2.2.60:45719)
0619 14:32:30.850214 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:30.850218 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:30.850231 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:32:30.850287 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:30.850289 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:30.850337 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:30.850438 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:32:30.850685 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.851210 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.852685 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.853245 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.853371 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:30.853519 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:30.853613 (+ 94us) server_negotiation.cc:299] Negotiation successful
0619 14:32:30.853687 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 14:32:30.854606 8974 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260619 14:32:30.855067 9003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf27644a9d194a898a1d93d6821e8b38. Latest consensus state: current_term: 1 leader_uuid: "cf27644a9d194a898a1d93d6821e8b38" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cf27644a9d194a898a1d93d6821e8b38" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 39059 } } peers { permanent_uuid: "e36baac4eca2403197084187e1360c08" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 36895 } attrs { promote: false } } peers { permanent_uuid: "03d127796b8d4c3286344b40cff82093" member_type: NON_VOTER last_known_addr { host: "127.2.2.60" port: 45719 } attrs { promote: true } } }
I20260619 14:32:30.855228 9003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093 [sys.catalog]: This master's current role is: LEARNER
I20260619 14:32:30.859748 9151 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.2:34039
I20260619 14:32:30.859786 9279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.2:34039 every 8 connection(s)
I20260619 14:32:30.860827 9151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
I20260619 14:32:30.863876 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 9151
I20260619 14:32:30.864048 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 8508
I20260619 14:32:30.872100 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.3:41487
--local_ip_for_outbound_sockets=127.2.2.3
--tserver_master_addrs=127.2.2.62:39059
--webserver_port=41245
--webserver_interface=127.2.2.3
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/krb5kdc/oryx_127.2.2.3.keytab
--principal=oryx/127.2.2.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:45229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.2.62:39059,127.2.2.61:36895
--tserver_master_addrs=127.2.2.62:39059,127.2.2.61:36895,127.2.2.60:45719 with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:32:30 dist-test-slave-70rm krb5kdc[8133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879550, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.2.2@KRBTEST.COM for oryx/127.2.2.61@KRBTEST.COM
Jun 19 14:32:30 dist-test-slave-70rm krb5kdc[8133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879550, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.2.2@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM
Jun 19 14:32:30 dist-test-slave-70rm krb5kdc[8133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879550, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.2.2@KRBTEST.COM for oryx/127.2.2.60@KRBTEST.COM
I20260619 14:32:30.880590 8901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:30.864354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:43921 (local address 127.2.2.61:36895)
0619 14:32:30.864793 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:30.864798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:30.864817 (+ 19us) server_negotiation.cc:407] Connection header received
0619 14:32:30.865713 (+ 896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:30.865719 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:30.865786 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:30.865885 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:30.867590 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.868416 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.870846 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.871200 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.871280 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:30.875440 (+ 4160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:30.875444 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:30.875447 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:30.875480 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:30.877708 (+ 2228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:30.878545 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:30.878548 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:30.878550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:30.878647 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:30.879135 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:30.879138 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:30.879139 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:30.879379 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:30.879463 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:30.880278 (+ 815us) server_negotiation.cc:299] Negotiation successful
0619 14:32:30.880416 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351}
I20260619 14:32:30.880591 8217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:30.862967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:56081 (local address 127.2.2.62:39059)
0619 14:32:30.863095 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:30.863099 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:30.865477 (+ 2378us) server_negotiation.cc:407] Connection header received
0619 14:32:30.866033 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:30.866039 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:30.866097 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:30.866216 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:30.867603 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.868416 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.870845 (+ 2429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.871200 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.871280 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:30.875853 (+ 4573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:30.875859 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:30.875868 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:30.875903 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:30.877945 (+ 2042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:30.878545 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:30.878548 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:30.878550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:30.878647 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:30.879130 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:30.879134 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:30.879136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:30.879379 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:30.879463 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:30.880278 (+ 815us) server_negotiation.cc:299] Negotiation successful
0619 14:32:30.880416 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 14:32:30.880692 9147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:30.862822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:44903 (local address 127.2.2.60:45719)
0619 14:32:30.862988 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:30.862995 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:30.866489 (+ 3494us) server_negotiation.cc:407] Connection header received
0619 14:32:30.866589 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:30.866594 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:30.866654 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:30.866734 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:30.867598 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.868495 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.870844 (+ 2349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.871282 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.871378 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:30.876047 (+ 4669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:30.876053 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:30.876056 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:30.876089 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:30.878244 (+ 2155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:30.878827 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:30.878834 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:30.878839 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:30.878936 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:30.879280 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:30.879286 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:30.879289 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:30.879542 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:30.879656 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:30.880295 (+ 639us) server_negotiation.cc:299] Negotiation successful
0619 14:32:30.880450 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 14:32:30.882650 9284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:30.863287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.60:45719 (local address 127.2.2.2:44903)
0619 14:32:30.866365 (+ 3078us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:30.866382 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:30.866500 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:30.866872 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:30.866878 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:30.867101 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:30.867409 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:30.867433 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.868658 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.868665 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:30.870668 (+ 2003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:30.870683 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.871533 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.871569 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:30.873674 (+ 2105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:30.874251 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:30.874271 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:30.875865 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:30.878391 (+ 2526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:30.878396 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:30.878401 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:30.878691 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:30.879058 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:30.879061 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:30.879064 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:30.879164 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:30.879722 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:30.879727 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:30.879982 (+ 255us) client_negotiation.cc:769] Sending connection context
0619 14:32:30.880836 (+ 854us) client_negotiation.cc:240] Negotiation successful
0619 14:32:30.881465 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":2369,"spinlock_wait_cycles":82048,"thread_start_us":152,"threads_started":1}
I20260619 14:32:30.882946 9285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:30.863426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.61:36895 (local address 127.2.2.2:43921)
0619 14:32:30.863870 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:30.863925 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:30.865484 (+ 1559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:30.866047 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:30.866055 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:30.866649 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:30.867411 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:30.867430 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.868605 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.868609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:30.870674 (+ 2065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:30.870685 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.871436 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.871503 (+ 67us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:30.871908 (+ 405us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:30.872975 (+ 1067us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:30.872987 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:30.875234 (+ 2247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:30.877897 (+ 2663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:30.877904 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:30.877924 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:30.878418 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:30.878771 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:30.878774 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:30.878776 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:30.879014 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:30.879530 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:30.879537 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:30.879968 (+ 431us) client_negotiation.cc:769] Sending connection context
0619 14:32:30.880836 (+ 868us) client_negotiation.cc:240] Negotiation successful
0619 14:32:30.881344 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":681,"spinlock_wait_cycles":29056,"thread_start_us":187,"threads_started":1}
I20260619 14:32:30.882674 9286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:30.864347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:39059 (local address 127.2.2.2:56081)
0619 14:32:30.865351 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:30.865366 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:30.865484 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:30.866633 (+ 1149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:30.866636 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:30.866801 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:30.867422 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:30.867430 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.868605 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.868609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:30.870676 (+ 2067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:30.870685 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:30.871436 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:30.871503 (+ 67us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:30.871908 (+ 405us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:30.872771 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:30.872801 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:30.875670 (+ 2869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:30.878065 (+ 2395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:30.878068 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:30.878070 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:30.878418 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:30.878771 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:30.878774 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:30.878776 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:30.879014 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:30.879530 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:30.879537 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:30.879968 (+ 431us) client_negotiation.cc:769] Sending connection context
0619 14:32:30.881244 (+ 1276us) client_negotiation.cc:240] Negotiation successful
0619 14:32:30.881542 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":929,"mutex_wait_us":940,"spinlock_wait_cycles":75520,"thread_start_us":95,"threads_started":1}
I20260619 14:32:30.883991 9281 heartbeater.cc:347] Connected to a master server at 127.2.2.60:45719
I20260619 14:32:30.884035 9280 heartbeater.cc:347] Connected to a master server at 127.2.2.62:39059
I20260619 14:32:30.884235 9281 heartbeater.cc:464] Registering TS with master...
I20260619 14:32:30.884276 9282 heartbeater.cc:347] Connected to a master server at 127.2.2.61:36895
I20260619 14:32:30.884235 9280 heartbeater.cc:464] Registering TS with master...
I20260619 14:32:30.884347 9282 heartbeater.cc:464] Registering TS with master...
I20260619 14:32:30.884853 9281 heartbeater.cc:511] Master 127.2.2.60:45719 requested a full tablet report, sending...
I20260619 14:32:30.884855 9280 heartbeater.cc:511] Master 127.2.2.62:39059 requested a full tablet report, sending...
I20260619 14:32:30.884853 9282 heartbeater.cc:511] Master 127.2.2.61:36895 requested a full tablet report, sending...
I20260619 14:32:30.885967 8174 ts_manager.cc:194] Re-registered known tserver with Master: 3cc7f187edfc4edabbb95a0f1bdce56a (127.2.2.2:34039)
I20260619 14:32:30.886039 8776 ts_manager.cc:194] Registered new tserver with Master: 3cc7f187edfc4edabbb95a0f1bdce56a (127.2.2.2:34039)
I20260619 14:32:30.886173 8964 ts_manager.cc:194] Registered new tserver with Master: 3cc7f187edfc4edabbb95a0f1bdce56a (127.2.2.2:34039)
I20260619 14:32:30.886785 8174 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.2.2@KRBTEST.COM'} at 127.2.2.2:56081
W20260619 14:32:30.996376 9287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:30.996697 9287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:30.996778 9287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:31.000989 9287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:31.001086 9287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:31.001189 9287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.3
I20260619 14:32:31.007052 9287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:45229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/krb5kdc/oryx_127.2.2.3.keytab
--principal=oryx/127.2.2.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.3:41487
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.2.3
--webserver_port=41245
--tserver_master_addrs=127.2.2.62:39059,127.2.2.61:36895,127.2.2.60:45719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.3
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:32:31.008212 9287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:31.009281 9287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:32:31.019665 9293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:31.019658 9292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:31.019697 9295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:31.020697 9287 server_base.cc:1034] running on GCE node
I20260619 14:32:31.021170 9287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:31.021854 9287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:31.023013 9287 hybrid_clock.cc:648] HybridClock initialized: now 1781879551022985 us; error 53 us; skew 500 ppm
Jun 19 14:32:31 dist-test-slave-70rm krb5kdc[8133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879551, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.2.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:32:31.025940 9287 init.cc:377] Logged in from keytab as oryx/127.2.2.3@KRBTEST.COM (short username oryx)
I20260619 14:32:31.026986 9287 webserver.cc:460] Webserver started at http://127.2.2.3:41245/ using document root <none> and password file <none>
I20260619 14:32:31.027590 9287 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:31.027667 9287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:31.031818 9287 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260619 14:32:31.034068 9302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:31.035491 9287 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 14:32:31.035609 9287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-2/wal
uuid: "c0f8745082c44f37a47d562686cf2780"
format_stamp: "Formatted at 2026-06-19 14:32:29 on dist-test-slave-70rm"
server_key: "2131872fe254872417e2d0c4d2b6aa76"
server_key_iv: "984966f9fa528722dde61a9010d54eae"
server_key_version: "encryptionkey@0"
I20260619 14:32:31.036259 9287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:31.058408 9287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:31.062256 9287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:32:31.062427 9287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:31.063009 9287 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:32:31.063869 9287 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:32:31.063915 9287 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:31.063956 9287 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:32:31.063971 9287 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:31.073122 9287 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.3:41487
I20260619 14:32:31.073169 9415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.3:41487 every 8 connection(s)
I20260619 14:32:31.074314 9287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
I20260619 14:32:31.080210 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 9287
Jun 19 14:32:31 dist-test-slave-70rm krb5kdc[8133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879551, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.2.3@KRBTEST.COM for oryx/127.2.2.60@KRBTEST.COM
Jun 19 14:32:31 dist-test-slave-70rm krb5kdc[8133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879551, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.2.3@KRBTEST.COM for oryx/127.2.2.61@KRBTEST.COM
Jun 19 14:32:31 dist-test-slave-70rm krb5kdc[8133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879551, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.2.3@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM
I20260619 14:32:31.094110 8901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:31.076376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.3:48467 (local address 127.2.2.61:36895)
0619 14:32:31.076632 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:31.076637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:31.078114 (+ 1477us) server_negotiation.cc:407] Connection header received
0619 14:32:31.079196 (+ 1082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:31.079198 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:31.079242 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:31.079339 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:31.080940 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:31.081706 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:31.083722 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:31.084074 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:31.084170 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:31.088400 (+ 4230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:31.088404 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:31.088406 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:31.088434 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:31.091206 (+ 2772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:31.092640 (+ 1434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:31.092655 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:31.092657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:31.092751 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:31.093091 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:31.093098 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:31.093102 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:31.093331 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:31.093418 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:31.093791 (+ 373us) server_negotiation.cc:299] Negotiation successful
0619 14:32:31.093938 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171}
I20260619 14:32:31.094574 9147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:31.076502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.3:54991 (local address 127.2.2.60:45719)
0619 14:32:31.076717 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:31.076721 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:31.078051 (+ 1330us) server_negotiation.cc:407] Connection header received
0619 14:32:31.078912 (+ 861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:31.078916 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:31.078977 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:31.079054 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:31.080946 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:31.081782 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:31.083727 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:31.084074 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:31.084178 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:31.088437 (+ 4259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:31.088442 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:31.088446 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:31.088480 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:31.090837 (+ 2357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:31.091819 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:31.091824 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:31.091827 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:31.091924 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:31.092346 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:31.092350 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:31.092352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:31.092564 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:31.092652 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:31.094261 (+ 1609us) server_negotiation.cc:299] Negotiation successful
0619 14:32:31.094385 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153}
I20260619 14:32:31.094978 9421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:31.076871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.61:36895 (local address 127.2.2.3:48467)
0619 14:32:31.077984 (+ 1113us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:31.078024 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:31.078563 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:31.079355 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:31.079359 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:31.080046 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:31.080753 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:31.080774 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:31.081899 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:31.081902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:31.083584 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:31.083593 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:31.084308 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:31.084360 (+ 52us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:31.085077 (+ 717us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:31.086067 (+ 990us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:31.086083 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:31.088224 (+ 2141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:31.091366 (+ 3142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:31.091370 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:31.091373 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:31.092519 (+ 1146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:31.092873 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:31.092878 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:31.092885 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:31.092978 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:31.093481 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:31.093485 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:31.093584 (+ 99us) client_negotiation.cc:769] Sending connection context
0619 14:32:31.093924 (+ 340us) client_negotiation.cc:240] Negotiation successful
0619 14:32:31.094054 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1017,"mutex_wait_us":810,"spinlock_wait_cycles":15616,"thread_start_us":150,"threads_started":1}
I20260619 14:32:31.095822 9422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:31.076872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.60:45719 (local address 127.2.2.3:54991)
0619 14:32:31.077932 (+ 1060us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:31.077946 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:31.078548 (+ 602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:31.079167 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:31.079177 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:31.079884 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:31.080760 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:31.080774 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:31.081934 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:31.081937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:31.083577 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:31.083588 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:31.084297 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:31.084360 (+ 63us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:31.084664 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:31.085373 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:31.085396 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:31.088227 (+ 2831us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:31.091189 (+ 2962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:31.091201 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:31.091225 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:31.091692 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:31.092037 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:31.092043 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:31.092046 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:31.092244 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:31.092717 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:31.092724 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:31.093103 (+ 379us) client_negotiation.cc:769] Sending connection context
0619 14:32:31.093328 (+ 225us) client_negotiation.cc:240] Negotiation successful
0619 14:32:31.093728 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":972,"mutex_wait_us":310,"spinlock_wait_cycles":40192,"thread_start_us":166,"threads_started":1}
I20260619 14:32:31.096922 9427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:31.082367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37430 (local address 127.2.2.61:36895)
0619 14:32:31.083818 (+ 1451us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:31.083823 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:31.083836 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:32:31.084247 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:31.084250 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:31.084294 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:31.084372 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:31.086441 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:31.087104 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:31.088302 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:31.088664 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:31.088769 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:31.090714 (+ 1945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:31.090718 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:31.090721 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:31.090749 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:31.094565 (+ 3816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:31.095041 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:31.095044 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:31.095046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:31.095125 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:31.095440 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:31.095443 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:31.095445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:31.095668 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:31.095819 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:31.096617 (+ 798us) server_negotiation.cc:299] Negotiation successful
0619 14:32:31.096750 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1382,"thread_start_us":88,"threads_started":1}
I20260619 14:32:31.096937 9417 heartbeater.cc:347] Connected to a master server at 127.2.2.60:45719
I20260619 14:32:31.097208 9417 heartbeater.cc:464] Registering TS with master...
I20260619 14:32:31.097698 8217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:31.076908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.3:60683 (local address 127.2.2.62:39059)
0619 14:32:31.077014 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:31.077018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:31.077594 (+ 576us) server_negotiation.cc:407] Connection header received
0619 14:32:31.078794 (+ 1200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:31.078798 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:31.078863 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:31.078953 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:31.080947 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:31.081782 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:31.084638 (+ 2856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:31.084989 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:31.085084 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:31.091930 (+ 6846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:31.091935 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:31.091938 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:31.091967 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:31.093716 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:31.096335 (+ 2619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:31.096338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:31.096340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:31.096431 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:31.096777 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:31.096780 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:31.096782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:31.096986 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:31.097073 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:31.097407 (+ 334us) server_negotiation.cc:299] Negotiation successful
0619 14:32:31.097527 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 14:32:31.097816 9417 heartbeater.cc:511] Master 127.2.2.60:45719 requested a full tablet report, sending...
I20260619 14:32:31.097950 9428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:31.082474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56018 (local address 127.2.2.60:45719)
0619 14:32:31.083677 (+ 1203us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:31.083683 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:31.084183 (+ 500us) server_negotiation.cc:407] Connection header received
0619 14:32:31.084584 (+ 401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:31.084589 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:31.084643 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:31.084716 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:31.085638 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:31.086451 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:31.088909 (+ 2458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:31.089278 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:31.089368 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:31.092929 (+ 3561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:31.092932 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:31.092935 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:31.092958 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:31.094681 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:31.095188 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:31.095192 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:31.095194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:31.095279 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:31.096682 (+ 1403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:31.096686 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:31.096687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:31.096895 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:31.097020 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:31.097665 (+ 645us) server_negotiation.cc:299] Negotiation successful
0619 14:32:31.097792 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1116,"thread_start_us":64,"threads_started":1}
I20260619 14:32:31.098204 9420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:31.076872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:39059 (local address 127.2.2.3:60683)
0619 14:32:31.077417 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:31.077471 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:31.078548 (+ 1077us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:31.079167 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:31.079177 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:31.079884 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:31.080758 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:31.080773 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:31.081934 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:31.081937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:31.084494 (+ 2557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:31.084505 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:31.085250 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:31.085290 (+ 40us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:31.085383 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:31.085841 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:31.085855 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:31.091774 (+ 5919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:31.095971 (+ 4197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:31.095977 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:31.095983 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:31.096225 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:31.096553 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:31.096556 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:31.096558 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:31.096666 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:31.097129 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:31.097131 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:31.097227 (+ 96us) client_negotiation.cc:769] Sending connection context
0619 14:32:31.097827 (+ 600us) client_negotiation.cc:240] Negotiation successful
0619 14:32:31.097982 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":98,"spinlock_wait_cycles":5632,"thread_start_us":143,"threads_started":1}
I20260619 14:32:31.099022 9416 heartbeater.cc:347] Connected to a master server at 127.2.2.62:39059
I20260619 14:32:31.099102 9416 heartbeater.cc:464] Registering TS with master...
I20260619 14:32:31.099145 9418 heartbeater.cc:347] Connected to a master server at 127.2.2.61:36895
I20260619 14:32:31.099195 9418 heartbeater.cc:464] Registering TS with master...
I20260619 14:32:31.099272 9416 heartbeater.cc:511] Master 127.2.2.62:39059 requested a full tablet report, sending...
I20260619 14:32:31.099294 9418 heartbeater.cc:511] Master 127.2.2.61:36895 requested a full tablet report, sending...
I20260619 14:32:31.099936 8174 ts_manager.cc:194] Re-registered known tserver with Master: c0f8745082c44f37a47d562686cf2780 (127.2.2.3:41487)
I20260619 14:32:31.099997 8964 ts_manager.cc:194] Registered new tserver with Master: c0f8745082c44f37a47d562686cf2780 (127.2.2.3:41487)
I20260619 14:32:31.100075 8776 ts_manager.cc:194] Registered new tserver with Master: c0f8745082c44f37a47d562686cf2780 (127.2.2.3:41487)
I20260619 14:32:31.100682 8174 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.2.3@KRBTEST.COM'} at 127.2.2.3:60683
I20260619 14:32:31.103605 9430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:31.082921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37816 (local address 127.2.2.62:39059)
0619 14:32:31.083509 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:31.083521 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:31.087503 (+ 3982us) server_negotiation.cc:407] Connection header received
0619 14:32:31.087663 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:31.087667 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:31.087726 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:31.087819 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:31.089902 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:31.090806 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:31.095413 (+ 4607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:31.095768 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:31.095847 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:31.099856 (+ 4009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:31.099860 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:31.099862 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:31.099889 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:31.101648 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:31.102269 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:31.102273 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:31.102274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:31.102368 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:31.102702 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:31.102705 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:31.102707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:31.102894 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:31.102999 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:31.103258 (+ 259us) server_negotiation.cc:299] Negotiation successful
0619 14:32:31.103363 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":97,"threads_started":1}
I20260619 14:32:31.106813 8174 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37816:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:32:31.109936 8174 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:32:31.113147 8936 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf27644a9d194a898a1d93d6821e8b38 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cf27644a9d194a898a1d93d6821e8b38" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cf27644a9d194a898a1d93d6821e8b38" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 39059 } } peers { permanent_uuid: "e36baac4eca2403197084187e1360c08" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 36895 } attrs { promote: false } } peers { permanent_uuid: "03d127796b8d4c3286344b40cff82093" member_type: NON_VOTER last_known_addr { host: "127.2.2.60" port: 45719 } attrs { promote: true } } }
I20260619 14:32:31.113333 8936 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf27644a9d194a898a1d93d6821e8b38 [sys.catalog]: This master's current role is: LEADER
I20260619 14:32:31.116758 9435 raft_consensus.cc:1050] T 00000000000000000000000000000000 P cf27644a9d194a898a1d93d6821e8b38: attempting to promote NON_VOTER 03d127796b8d4c3286344b40cff82093 to VOTER
I20260619 14:32:31.117367 9435 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf27644a9d194a898a1d93d6821e8b38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cf27644a9d194a898a1d93d6821e8b38" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 39059 } } peers { permanent_uuid: "e36baac4eca2403197084187e1360c08" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 36895 } attrs { promote: false } } peers { permanent_uuid: "03d127796b8d4c3286344b40cff82093" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 45719 } attrs { promote: false } }
I20260619 14:32:31.118533 8789 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [term 1 FOLLOWER]: Refusing update from remote peer cf27644a9d194a898a1d93d6821e8b38: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260619 14:32:31.118759 8973 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093 [term 1 LEARNER]: Refusing update from remote peer cf27644a9d194a898a1d93d6821e8b38: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260619 14:32:31.118963 8936 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cf27644a9d194a898a1d93d6821e8b38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e36baac4eca2403197084187e1360c08" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 36895 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260619 14:32:31.119256 9434 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cf27644a9d194a898a1d93d6821e8b38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03d127796b8d4c3286344b40cff82093" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 45719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260619 14:32:31.120820 8936 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cf27644a9d194a898a1d93d6821e8b38 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 03d127796b8d4c3286344b40cff82093 (127.2.2.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cf27644a9d194a898a1d93d6821e8b38" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 39059 } } peers { permanent_uuid: "e36baac4eca2403197084187e1360c08" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 36895 } attrs { promote: false } } peers { permanent_uuid: "03d127796b8d4c3286344b40cff82093" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 45719 } attrs { promote: false } } }
I20260619 14:32:31.122285 8936 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf27644a9d194a898a1d93d6821e8b38 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "cf27644a9d194a898a1d93d6821e8b38" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cf27644a9d194a898a1d93d6821e8b38" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 39059 } } peers { permanent_uuid: "e36baac4eca2403197084187e1360c08" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 36895 } attrs { promote: false } } peers { permanent_uuid: "03d127796b8d4c3286344b40cff82093" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 45719 } attrs { promote: false } } }
I20260619 14:32:31.122334 9435 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf27644a9d194a898a1d93d6821e8b38 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cf27644a9d194a898a1d93d6821e8b38" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cf27644a9d194a898a1d93d6821e8b38" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 39059 } } peers { permanent_uuid: "e36baac4eca2403197084187e1360c08" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 36895 } attrs { promote: false } } peers { permanent_uuid: "03d127796b8d4c3286344b40cff82093" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 45719 } attrs { promote: false } } }
I20260619 14:32:31.122452 9435 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf27644a9d194a898a1d93d6821e8b38 [sys.catalog]: This master's current role is: LEADER
I20260619 14:32:31.122591 9435 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf27644a9d194a898a1d93d6821e8b38 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cf27644a9d194a898a1d93d6821e8b38" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cf27644a9d194a898a1d93d6821e8b38" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 39059 } } peers { permanent_uuid: "e36baac4eca2403197084187e1360c08" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 36895 } attrs { promote: false } } peers { permanent_uuid: "03d127796b8d4c3286344b40cff82093" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 45719 } attrs { promote: false } } }
I20260619 14:32:31.122401 8936 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf27644a9d194a898a1d93d6821e8b38 [sys.catalog]: This master's current role is: LEADER
I20260619 14:32:31.122680 9435 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf27644a9d194a898a1d93d6821e8b38 [sys.catalog]: This master's current role is: LEADER
I20260619 14:32:31.122812 8789 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 03d127796b8d4c3286344b40cff82093 (127.2.2.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cf27644a9d194a898a1d93d6821e8b38" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 39059 } } peers { permanent_uuid: "e36baac4eca2403197084187e1360c08" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 36895 } attrs { promote: false } } peers { permanent_uuid: "03d127796b8d4c3286344b40cff82093" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 45719 } attrs { promote: false } } }
I20260619 14:32:31.123159 8973 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 03d127796b8d4c3286344b40cff82093 (127.2.2.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cf27644a9d194a898a1d93d6821e8b38" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 39059 } } peers { permanent_uuid: "e36baac4eca2403197084187e1360c08" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 36895 } attrs { promote: false } } peers { permanent_uuid: "03d127796b8d4c3286344b40cff82093" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 45719 } attrs { promote: false } } }
I20260619 14:32:31.124878 9443 sys_catalog.cc:455] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cf27644a9d194a898a1d93d6821e8b38" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cf27644a9d194a898a1d93d6821e8b38" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 39059 } } peers { permanent_uuid: "e36baac4eca2403197084187e1360c08" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 36895 } attrs { promote: false } } peers { permanent_uuid: "03d127796b8d4c3286344b40cff82093" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 45719 } attrs { promote: false } } }
I20260619 14:32:31.125036 9443 sys_catalog.cc:458] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:32:31.125291 9003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cf27644a9d194a898a1d93d6821e8b38" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cf27644a9d194a898a1d93d6821e8b38" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 39059 } } peers { permanent_uuid: "e36baac4eca2403197084187e1360c08" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 36895 } attrs { promote: false } } peers { permanent_uuid: "03d127796b8d4c3286344b40cff82093" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 45719 } attrs { promote: false } } }
I20260619 14:32:31.125414 9003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:32:31.134605 8735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:31.128564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.1:45703 (local address 127.0.0.1:46652)
0619 14:32:31.128697 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:31.128713 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:31.128796 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:31.129863 (+ 1067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:31.129866 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:31.129886 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:32:31.130008 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:31.130016 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:31.131758 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:31.131762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:31.133013 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:31.133024 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:31.134165 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:31.134169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:31.134271 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:31.134301 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 14:32:31.134360 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 14:32:31.134435 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":5}
I20260619 14:32:31.134665 9444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:31.128835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46652 (local address 127.2.2.1:45703)
0619 14:32:31.129125 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:31.129144 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:31.129170 (+ 26us) server_negotiation.cc:407] Connection header received
0619 14:32:31.129255 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:31.129261 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:31.129689 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:31.130217 (+ 528us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:32:31.130428 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:31.131616 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:31.133182 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:31.134007 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:31.134171 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:31.134328 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:31.134394 (+ 66us) server_negotiation.cc:299] Negotiation successful
0619 14:32:31.134498 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":109,"threads_started":1}
I20260619 14:32:31.134933 9445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:31.129081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.2:34039 (local address 127.0.0.1:37478)
0619 14:32:31.129951 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:31.129964 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:31.130046 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:31.130606 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:31.130609 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:31.130624 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:32:31.130734 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:31.130744 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:31.132013 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:31.132017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:31.133204 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:31.133211 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:31.134441 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:31.134445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:31.134532 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:31.134555 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 14:32:31.134639 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 14:32:31.134721 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":814,"mutex_wait_us":29,"thread_start_us":107,"threads_started":1}
I20260619 14:32:31.135155 9447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:31.129470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49276 (local address 127.2.2.3:41487)
0619 14:32:31.129899 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:31.129912 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:31.130360 (+ 448us) server_negotiation.cc:407] Connection header received
0619 14:32:31.130467 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:31.130473 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:31.130676 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:31.130850 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:32:31.131155 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:31.131889 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:31.133089 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:31.133957 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:31.134032 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:31.134229 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:31.134556 (+ 327us) server_negotiation.cc:299] Negotiation successful
0619 14:32:31.134963 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":103,"threads_started":1}
I20260619 14:32:31.135427 9448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:31.129469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37478 (local address 127.2.2.2:34039)
0619 14:32:31.130106 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:31.130145 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:31.130169 (+ 24us) server_negotiation.cc:407] Connection header received
0619 14:32:31.130245 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:31.130250 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:31.130470 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:31.130639 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:32:31.130843 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:31.131897 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:31.133342 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:31.134179 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:31.134269 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:31.134379 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:31.135196 (+ 817us) server_negotiation.cc:299] Negotiation successful
0619 14:32:31.135295 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":100,"threads_started":1}
I20260619 14:32:31.135561 9446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:31.129102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.3:41487 (local address 127.0.0.1:49276)
0619 14:32:31.130251 (+ 1149us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:31.130263 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:31.130360 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:31.130974 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:31.130976 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:31.130989 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:32:31.131049 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:31.131053 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:31.132005 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:31.132011 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:31.132940 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:31.132954 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:31.134323 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:31.134329 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:31.134414 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:31.134433 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 14:32:31.135318 (+ 885us) client_negotiation.cc:240] Negotiation successful
0619 14:32:31.135399 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1089,"mutex_wait_us":65,"thread_start_us":88,"threads_started":1}
I20260619 14:32:31.138269 9350 tablet_service.cc:1467] Processing CreateTablet for tablet a43359647ee446a1b6cd04d6906bec71 (DEFAULT_TABLE table=test-table [id=84c43d500a5649c8b8db92ca873c025e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:32:31.138211 9214 tablet_service.cc:1467] Processing CreateTablet for tablet a43359647ee446a1b6cd04d6906bec71 (DEFAULT_TABLE table=test-table [id=84c43d500a5649c8b8db92ca873c025e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:32:31.139408 9350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a43359647ee446a1b6cd04d6906bec71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:31.139612 9077 tablet_service.cc:1467] Processing CreateTablet for tablet a43359647ee446a1b6cd04d6906bec71 (DEFAULT_TABLE table=test-table [id=84c43d500a5649c8b8db92ca873c025e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:32:31.140653 9077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a43359647ee446a1b6cd04d6906bec71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:31.141388 9214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a43359647ee446a1b6cd04d6906bec71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:31.149209 9450 tablet_bootstrap.cc:492] T a43359647ee446a1b6cd04d6906bec71 P c81bd9d0e5f44e0fa56ac4dcc2401a27: Bootstrap starting.
I20260619 14:32:31.150005 9449 tablet_bootstrap.cc:492] T a43359647ee446a1b6cd04d6906bec71 P c0f8745082c44f37a47d562686cf2780: Bootstrap starting.
I20260619 14:32:31.150825 9451 tablet_bootstrap.cc:492] T a43359647ee446a1b6cd04d6906bec71 P 3cc7f187edfc4edabbb95a0f1bdce56a: Bootstrap starting.
I20260619 14:32:31.151604 9450 tablet_bootstrap.cc:654] T a43359647ee446a1b6cd04d6906bec71 P c81bd9d0e5f44e0fa56ac4dcc2401a27: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:31.152165 9449 tablet_bootstrap.cc:654] T a43359647ee446a1b6cd04d6906bec71 P c0f8745082c44f37a47d562686cf2780: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:31.152493 9450 log.cc:824] T a43359647ee446a1b6cd04d6906bec71 P c81bd9d0e5f44e0fa56ac4dcc2401a27: Log is configured to *not* fsync() on all Append() calls
I20260619 14:32:31.153350 9451 tablet_bootstrap.cc:654] T a43359647ee446a1b6cd04d6906bec71 P 3cc7f187edfc4edabbb95a0f1bdce56a: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:31.153461 9449 log.cc:824] T a43359647ee446a1b6cd04d6906bec71 P c0f8745082c44f37a47d562686cf2780: Log is configured to *not* fsync() on all Append() calls
I20260619 14:32:31.154613 9451 log.cc:824] T a43359647ee446a1b6cd04d6906bec71 P 3cc7f187edfc4edabbb95a0f1bdce56a: Log is configured to *not* fsync() on all Append() calls
I20260619 14:32:31.155368 9450 tablet_bootstrap.cc:492] T a43359647ee446a1b6cd04d6906bec71 P c81bd9d0e5f44e0fa56ac4dcc2401a27: No bootstrap required, opened a new log
I20260619 14:32:31.155652 9450 ts_tablet_manager.cc:1397] T a43359647ee446a1b6cd04d6906bec71 P c81bd9d0e5f44e0fa56ac4dcc2401a27: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260619 14:32:31.157383 9451 tablet_bootstrap.cc:492] T a43359647ee446a1b6cd04d6906bec71 P 3cc7f187edfc4edabbb95a0f1bdce56a: No bootstrap required, opened a new log
I20260619 14:32:31.157644 9451 ts_tablet_manager.cc:1397] T a43359647ee446a1b6cd04d6906bec71 P 3cc7f187edfc4edabbb95a0f1bdce56a: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260619 14:32:31.160527 9450 raft_consensus.cc:348] T a43359647ee446a1b6cd04d6906bec71 P c81bd9d0e5f44e0fa56ac4dcc2401a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c81bd9d0e5f44e0fa56ac4dcc2401a27" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 45703 } } peers { permanent_uuid: "c0f8745082c44f37a47d562686cf2780" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 41487 } } peers { permanent_uuid: "3cc7f187edfc4edabbb95a0f1bdce56a" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 34039 } }
I20260619 14:32:31.160913 9450 raft_consensus.cc:374] T a43359647ee446a1b6cd04d6906bec71 P c81bd9d0e5f44e0fa56ac4dcc2401a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:31.160969 9450 raft_consensus.cc:729] T a43359647ee446a1b6cd04d6906bec71 P c81bd9d0e5f44e0fa56ac4dcc2401a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c81bd9d0e5f44e0fa56ac4dcc2401a27, State: Initialized, Role: FOLLOWER
I20260619 14:32:31.161542 9450 consensus_queue.cc:260] T a43359647ee446a1b6cd04d6906bec71 P c81bd9d0e5f44e0fa56ac4dcc2401a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c81bd9d0e5f44e0fa56ac4dcc2401a27" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 45703 } } peers { permanent_uuid: "c0f8745082c44f37a47d562686cf2780" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 41487 } } peers { permanent_uuid: "3cc7f187edfc4edabbb95a0f1bdce56a" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 34039 } }
I20260619 14:32:31.162314 9143 heartbeater.cc:503] Master 127.2.2.62:39059 was elected leader, sending a full tablet report...
I20260619 14:32:31.163703 9450 ts_tablet_manager.cc:1428] T a43359647ee446a1b6cd04d6906bec71 P c81bd9d0e5f44e0fa56ac4dcc2401a27: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260619 14:32:31.164167 9451 raft_consensus.cc:348] T a43359647ee446a1b6cd04d6906bec71 P 3cc7f187edfc4edabbb95a0f1bdce56a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c81bd9d0e5f44e0fa56ac4dcc2401a27" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 45703 } } peers { permanent_uuid: "c0f8745082c44f37a47d562686cf2780" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 41487 } } peers { permanent_uuid: "3cc7f187edfc4edabbb95a0f1bdce56a" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 34039 } }
I20260619 14:32:31.164453 9451 raft_consensus.cc:374] T a43359647ee446a1b6cd04d6906bec71 P 3cc7f187edfc4edabbb95a0f1bdce56a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:31.164501 9451 raft_consensus.cc:729] T a43359647ee446a1b6cd04d6906bec71 P 3cc7f187edfc4edabbb95a0f1bdce56a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cc7f187edfc4edabbb95a0f1bdce56a, State: Initialized, Role: FOLLOWER
I20260619 14:32:31.165042 9451 consensus_queue.cc:260] T a43359647ee446a1b6cd04d6906bec71 P 3cc7f187edfc4edabbb95a0f1bdce56a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c81bd9d0e5f44e0fa56ac4dcc2401a27" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 45703 } } peers { permanent_uuid: "c0f8745082c44f37a47d562686cf2780" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 41487 } } peers { permanent_uuid: "3cc7f187edfc4edabbb95a0f1bdce56a" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 34039 } }
I20260619 14:32:31.165594 9449 tablet_bootstrap.cc:492] T a43359647ee446a1b6cd04d6906bec71 P c0f8745082c44f37a47d562686cf2780: No bootstrap required, opened a new log
I20260619 14:32:31.165812 9449 ts_tablet_manager.cc:1397] T a43359647ee446a1b6cd04d6906bec71 P c0f8745082c44f37a47d562686cf2780: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.000s
I20260619 14:32:31.166170 9451 ts_tablet_manager.cc:1428] T a43359647ee446a1b6cd04d6906bec71 P 3cc7f187edfc4edabbb95a0f1bdce56a: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260619 14:32:31.166525 9280 heartbeater.cc:503] Master 127.2.2.62:39059 was elected leader, sending a full tablet report...
I20260619 14:32:31.170756 9449 raft_consensus.cc:348] T a43359647ee446a1b6cd04d6906bec71 P c0f8745082c44f37a47d562686cf2780 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c81bd9d0e5f44e0fa56ac4dcc2401a27" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 45703 } } peers { permanent_uuid: "c0f8745082c44f37a47d562686cf2780" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 41487 } } peers { permanent_uuid: "3cc7f187edfc4edabbb95a0f1bdce56a" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 34039 } }
I20260619 14:32:31.171135 9449 raft_consensus.cc:374] T a43359647ee446a1b6cd04d6906bec71 P c0f8745082c44f37a47d562686cf2780 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:31.171214 9449 raft_consensus.cc:729] T a43359647ee446a1b6cd04d6906bec71 P c0f8745082c44f37a47d562686cf2780 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f8745082c44f37a47d562686cf2780, State: Initialized, Role: FOLLOWER
I20260619 14:32:31.171777 9449 consensus_queue.cc:260] T a43359647ee446a1b6cd04d6906bec71 P c0f8745082c44f37a47d562686cf2780 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c81bd9d0e5f44e0fa56ac4dcc2401a27" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 45703 } } peers { permanent_uuid: "c0f8745082c44f37a47d562686cf2780" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 41487 } } peers { permanent_uuid: "3cc7f187edfc4edabbb95a0f1bdce56a" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 34039 } }
I20260619 14:32:31.172955 9449 ts_tablet_manager.cc:1428] T a43359647ee446a1b6cd04d6906bec71 P c0f8745082c44f37a47d562686cf2780: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260619 14:32:31.173955 9416 heartbeater.cc:503] Master 127.2.2.62:39059 was elected leader, sending a full tablet report...
I20260619 14:32:31.243173 9455 raft_consensus.cc:482] T a43359647ee446a1b6cd04d6906bec71 P c81bd9d0e5f44e0fa56ac4dcc2401a27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:32:31.243369 9455 raft_consensus.cc:504] T a43359647ee446a1b6cd04d6906bec71 P c81bd9d0e5f44e0fa56ac4dcc2401a27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c81bd9d0e5f44e0fa56ac4dcc2401a27" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 45703 } } peers { permanent_uuid: "c0f8745082c44f37a47d562686cf2780" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 41487 } } peers { permanent_uuid: "3cc7f187edfc4edabbb95a0f1bdce56a" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 34039 } }
I20260619 14:32:31.244530 9455 leader_election.cc:288] T a43359647ee446a1b6cd04d6906bec71 P c81bd9d0e5f44e0fa56ac4dcc2401a27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0f8745082c44f37a47d562686cf2780 (127.2.2.3:41487), 3cc7f187edfc4edabbb95a0f1bdce56a (127.2.2.2:34039)
I20260619 14:32:31.248487 9447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:31.244887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:35039 (local address 127.2.2.3:41487)
0619 14:32:31.245015 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:31.245020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:31.245109 (+ 89us) server_negotiation.cc:407] Connection header received
0619 14:32:31.245342 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:31.245347 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:31.245407 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:31.245464 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:32:31.245796 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:31.246375 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:31.247518 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:31.247905 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:31.247986 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:31.248080 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:31.248268 (+ 188us) server_negotiation.cc:299] Negotiation successful
0619 14:32:31.248356 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 14:32:31.248510 9459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:31.244861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.2:34039 (local address 127.2.2.1:44201)
0619 14:32:31.245159 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:31.245171 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:31.245277 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:31.245574 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:31.245577 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:31.245594 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:32:31.245683 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:31.245687 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:31.246462 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:31.246467 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:31.247385 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:31.247394 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:31.248080 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:31.248084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:31.248180 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:31.248196 (+ 16us) client_negotiation.cc:769] Sending connection context
0619 14:32:31.248293 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 14:32:31.248356 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":52,"thread_start_us":85,"threads_started":1}
I20260619 14:32:31.248518 9448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:31.245063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:44201 (local address 127.2.2.2:34039)
0619 14:32:31.245204 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:31.245209 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:31.245226 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:32:31.245394 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:31.245397 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:31.245438 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:31.245538 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:32:31.245825 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:31.246355 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:31.247514 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:31.247932 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:31.248055 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:31.248117 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:31.248326 (+ 209us) server_negotiation.cc:299] Negotiation successful
0619 14:32:31.248410 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 14:32:31.248512 9458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:31.244743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.3:41487 (local address 127.2.2.1:35039)
0619 14:32:31.245016 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:31.245032 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:31.245198 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:31.245504 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:31.245510 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:31.245531 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:32:31.245677 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:31.245683 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:31.246499 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:31.246503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:31.247389 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:31.247397 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:31.248027 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:31.248030 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:31.248118 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:31.248148 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 14:32:31.248221 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 14:32:31.248312 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":24,"thread_start_us":90,"threads_started":1}
I20260619 14:32:31.249234 9370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43359647ee446a1b6cd04d6906bec71" candidate_uuid: "c81bd9d0e5f44e0fa56ac4dcc2401a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f8745082c44f37a47d562686cf2780" is_pre_election: true
I20260619 14:32:31.249235 9233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43359647ee446a1b6cd04d6906bec71" candidate_uuid: "c81bd9d0e5f44e0fa56ac4dcc2401a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cc7f187edfc4edabbb95a0f1bdce56a" is_pre_election: true
I20260619 14:32:31.249547 9370 raft_consensus.cc:2454] T a43359647ee446a1b6cd04d6906bec71 P c0f8745082c44f37a47d562686cf2780 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c81bd9d0e5f44e0fa56ac4dcc2401a27 in term 0.
I20260619 14:32:31.249553 9233 raft_consensus.cc:2454] T a43359647ee446a1b6cd04d6906bec71 P 3cc7f187edfc4edabbb95a0f1bdce56a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c81bd9d0e5f44e0fa56ac4dcc2401a27 in term 0.
I20260619 14:32:31.250066 9032 leader_election.cc:302] T a43359647ee446a1b6cd04d6906bec71 P c81bd9d0e5f44e0fa56ac4dcc2401a27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3cc7f187edfc4edabbb95a0f1bdce56a, c0f8745082c44f37a47d562686cf2780, c81bd9d0e5f44e0fa56ac4dcc2401a27; no voters:
I20260619 14:32:31.250412 9455 raft_consensus.cc:2781] T a43359647ee446a1b6cd04d6906bec71 P c81bd9d0e5f44e0fa56ac4dcc2401a27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:32:31.250496 9455 raft_consensus.cc:482] T a43359647ee446a1b6cd04d6906bec71 P c81bd9d0e5f44e0fa56ac4dcc2401a27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:32:31.250540 9455 raft_consensus.cc:3037] T a43359647ee446a1b6cd04d6906bec71 P c81bd9d0e5f44e0fa56ac4dcc2401a27 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:31.251576 9455 raft_consensus.cc:504] T a43359647ee446a1b6cd04d6906bec71 P c81bd9d0e5f44e0fa56ac4dcc2401a27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c81bd9d0e5f44e0fa56ac4dcc2401a27" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 45703 } } peers { permanent_uuid: "c0f8745082c44f37a47d562686cf2780" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 41487 } } peers { permanent_uuid: "3cc7f187edfc4edabbb95a0f1bdce56a" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 34039 } }
I20260619 14:32:31.251936 9455 leader_election.cc:288] T a43359647ee446a1b6cd04d6906bec71 P c81bd9d0e5f44e0fa56ac4dcc2401a27 [CANDIDATE]: Term 1 election: Requested vote from peers c0f8745082c44f37a47d562686cf2780 (127.2.2.3:41487), 3cc7f187edfc4edabbb95a0f1bdce56a (127.2.2.2:34039)
I20260619 14:32:31.252324 9370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43359647ee446a1b6cd04d6906bec71" candidate_uuid: "c81bd9d0e5f44e0fa56ac4dcc2401a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f8745082c44f37a47d562686cf2780"
I20260619 14:32:31.252331 9233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43359647ee446a1b6cd04d6906bec71" candidate_uuid: "c81bd9d0e5f44e0fa56ac4dcc2401a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cc7f187edfc4edabbb95a0f1bdce56a"
I20260619 14:32:31.252451 9370 raft_consensus.cc:3037] T a43359647ee446a1b6cd04d6906bec71 P c0f8745082c44f37a47d562686cf2780 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:31.252451 9233 raft_consensus.cc:3037] T a43359647ee446a1b6cd04d6906bec71 P 3cc7f187edfc4edabbb95a0f1bdce56a [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:31.253736 9233 raft_consensus.cc:2454] T a43359647ee446a1b6cd04d6906bec71 P 3cc7f187edfc4edabbb95a0f1bdce56a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c81bd9d0e5f44e0fa56ac4dcc2401a27 in term 1.
I20260619 14:32:31.253741 9370 raft_consensus.cc:2454] T a43359647ee446a1b6cd04d6906bec71 P c0f8745082c44f37a47d562686cf2780 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c81bd9d0e5f44e0fa56ac4dcc2401a27 in term 1.
I20260619 14:32:31.254014 9032 leader_election.cc:302] T a43359647ee446a1b6cd04d6906bec71 P c81bd9d0e5f44e0fa56ac4dcc2401a27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0f8745082c44f37a47d562686cf2780, c81bd9d0e5f44e0fa56ac4dcc2401a27; no voters:
I20260619 14:32:31.254220 9455 raft_consensus.cc:2781] T a43359647ee446a1b6cd04d6906bec71 P c81bd9d0e5f44e0fa56ac4dcc2401a27 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:32:31.254467 9455 raft_consensus.cc:686] T a43359647ee446a1b6cd04d6906bec71 P c81bd9d0e5f44e0fa56ac4dcc2401a27 [term 1 LEADER]: Becoming Leader. State: Replica: c81bd9d0e5f44e0fa56ac4dcc2401a27, State: Running, Role: LEADER
I20260619 14:32:31.254834 9455 consensus_queue.cc:237] T a43359647ee446a1b6cd04d6906bec71 P c81bd9d0e5f44e0fa56ac4dcc2401a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c81bd9d0e5f44e0fa56ac4dcc2401a27" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 45703 } } peers { permanent_uuid: "c0f8745082c44f37a47d562686cf2780" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 41487 } } peers { permanent_uuid: "3cc7f187edfc4edabbb95a0f1bdce56a" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 34039 } }
I20260619 14:32:31.258103 8178 catalog_manager.cc:5495] T a43359647ee446a1b6cd04d6906bec71 P c81bd9d0e5f44e0fa56ac4dcc2401a27 reported cstate change: term changed from 0 to 1, leader changed from <none> to c81bd9d0e5f44e0fa56ac4dcc2401a27 (127.2.2.1). New cstate: current_term: 1 leader_uuid: "c81bd9d0e5f44e0fa56ac4dcc2401a27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c81bd9d0e5f44e0fa56ac4dcc2401a27" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 45703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f8745082c44f37a47d562686cf2780" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 41487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cc7f187edfc4edabbb95a0f1bdce56a" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 34039 } health_report { overall_health: UNKNOWN } } }
I20260619 14:32:31.293828 9444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:31.290175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46668 (local address 127.2.2.1:45703)
0619 14:32:31.290311 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:31.290318 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:31.290335 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:32:31.290435 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:31.290438 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:31.290495 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:31.290617 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:32:31.290867 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:31.291413 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:31.292296 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:31.292519 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:31.292632 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:31.292957 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:32:31.293411 (+ 454us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:32:31.293553 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:31.293614 (+ 61us) server_negotiation.cc:299] Negotiation successful
0619 14:32:31.293674 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 14:32:31.300549 9233 raft_consensus.cc:1261] T a43359647ee446a1b6cd04d6906bec71 P 3cc7f187edfc4edabbb95a0f1bdce56a [term 1 FOLLOWER]: Refusing update from remote peer c81bd9d0e5f44e0fa56ac4dcc2401a27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:32:31.300549 9370 raft_consensus.cc:1261] T a43359647ee446a1b6cd04d6906bec71 P c0f8745082c44f37a47d562686cf2780 [term 1 FOLLOWER]: Refusing update from remote peer c81bd9d0e5f44e0fa56ac4dcc2401a27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:32:31.301335 9460 consensus_queue.cc:1035] T a43359647ee446a1b6cd04d6906bec71 P c81bd9d0e5f44e0fa56ac4dcc2401a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cc7f187edfc4edabbb95a0f1bdce56a" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 34039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:32:31.301592 9455 consensus_queue.cc:1035] T a43359647ee446a1b6cd04d6906bec71 P c81bd9d0e5f44e0fa56ac4dcc2401a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f8745082c44f37a47d562686cf2780" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 41487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:32:31.312060 9465 mvcc.cc:205] Tried to move back new op lower bound from 7298578642118660096 to 7298578641943048192. Current Snapshot: MvccSnapshot[applied={T|T < 7298578642118660096 or (T in {7298578642118660096})}]
I20260619 14:32:31.313341 9464 mvcc.cc:205] Tried to move back new op lower bound from 7298578642118660096 to 7298578641943048192. Current Snapshot: MvccSnapshot[applied={T|T < 7298578642118660096}]
I20260619 14:32:31.320417 8174 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:32:31.320616 8174 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:32:31.324625 8174 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cf27644a9d194a898a1d93d6821e8b38: Sending DeleteTablet for 3 replicas of tablet a43359647ee446a1b6cd04d6906bec71
I20260619 14:32:31.325572 9077 tablet_service.cc:1514] Processing DeleteTablet for tablet a43359647ee446a1b6cd04d6906bec71 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:32:31 UTC) from {username='oryx', principal='oryx/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:46652
I20260619 14:32:31.325605 9350 tablet_service.cc:1514] Processing DeleteTablet for tablet a43359647ee446a1b6cd04d6906bec71 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:32:31 UTC) from {username='oryx', principal='oryx/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:49276
I20260619 14:32:31.326472 9214 tablet_service.cc:1514] Processing DeleteTablet for tablet a43359647ee446a1b6cd04d6906bec71 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:32:31 UTC) from {username='oryx', principal='oryx/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:37478
I20260619 14:32:31.327025 9475 tablet_replica.cc:330] T a43359647ee446a1b6cd04d6906bec71 P 3cc7f187edfc4edabbb95a0f1bdce56a: stopping tablet replica
I20260619 14:32:31.327426 9475 raft_consensus.cc:2229] T a43359647ee446a1b6cd04d6906bec71 P 3cc7f187edfc4edabbb95a0f1bdce56a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:32:31.327790 9475 raft_consensus.cc:2258] T a43359647ee446a1b6cd04d6906bec71 P 3cc7f187edfc4edabbb95a0f1bdce56a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:32:31.328009 9474 tablet_replica.cc:330] T a43359647ee446a1b6cd04d6906bec71 P c81bd9d0e5f44e0fa56ac4dcc2401a27: stopping tablet replica
I20260619 14:32:31.328296 9474 raft_consensus.cc:2229] T a43359647ee446a1b6cd04d6906bec71 P c81bd9d0e5f44e0fa56ac4dcc2401a27 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:32:31.328661 9474 raft_consensus.cc:2258] T a43359647ee446a1b6cd04d6906bec71 P c81bd9d0e5f44e0fa56ac4dcc2401a27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:32:31.329504 9475 ts_tablet_manager.cc:1905] T a43359647ee446a1b6cd04d6906bec71 P 3cc7f187edfc4edabbb95a0f1bdce56a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:32:31.329998 9474 ts_tablet_manager.cc:1905] T a43359647ee446a1b6cd04d6906bec71 P c81bd9d0e5f44e0fa56ac4dcc2401a27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:32:31.331048 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 8146
I20260619 14:32:31.333298 9475 ts_tablet_manager.cc:1918] T a43359647ee446a1b6cd04d6906bec71 P 3cc7f187edfc4edabbb95a0f1bdce56a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:32:31.333413 9475 log.cc:1196] T a43359647ee446a1b6cd04d6906bec71 P 3cc7f187edfc4edabbb95a0f1bdce56a: Deleting WAL directory at /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-1/wal/wals/a43359647ee446a1b6cd04d6906bec71
I20260619 14:32:31.333753 9475 ts_tablet_manager.cc:1939] T a43359647ee446a1b6cd04d6906bec71 P 3cc7f187edfc4edabbb95a0f1bdce56a: Deleting consensus metadata
I20260619 14:32:31.333741 9474 ts_tablet_manager.cc:1918] T a43359647ee446a1b6cd04d6906bec71 P c81bd9d0e5f44e0fa56ac4dcc2401a27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:32:31.333829 9474 log.cc:1196] T a43359647ee446a1b6cd04d6906bec71 P c81bd9d0e5f44e0fa56ac4dcc2401a27: Deleting WAL directory at /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-0/wal/wals/a43359647ee446a1b6cd04d6906bec71
I20260619 14:32:31.334178 9474 ts_tablet_manager.cc:1939] T a43359647ee446a1b6cd04d6906bec71 P c81bd9d0e5f44e0fa56ac4dcc2401a27: Deleting consensus metadata
I20260619 14:32:31.335253 9473 tablet_replica.cc:330] T a43359647ee446a1b6cd04d6906bec71 P c0f8745082c44f37a47d562686cf2780: stopping tablet replica
I20260619 14:32:31.335614 9473 raft_consensus.cc:2229] T a43359647ee446a1b6cd04d6906bec71 P c0f8745082c44f37a47d562686cf2780 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:32:31.335989 9473 raft_consensus.cc:2258] T a43359647ee446a1b6cd04d6906bec71 P c0f8745082c44f37a47d562686cf2780 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:32:31.337800 9473 ts_tablet_manager.cc:1905] T a43359647ee446a1b6cd04d6906bec71 P c0f8745082c44f37a47d562686cf2780: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 14:32:31.339129 9168 connection.cc:535] server connection from 127.0.0.1:37478 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260619 14:32:31.339613 9031 connection.cc:535] server connection from 127.0.0.1:46652 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 14:32:31.341778 9473 ts_tablet_manager.cc:1918] T a43359647ee446a1b6cd04d6906bec71 P c0f8745082c44f37a47d562686cf2780: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:32:31.341894 9473 log.cc:1196] T a43359647ee446a1b6cd04d6906bec71 P c0f8745082c44f37a47d562686cf2780: Deleting WAL directory at /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-2/wal/wals/a43359647ee446a1b6cd04d6906bec71
I20260619 14:32:31.342298 9473 ts_tablet_manager.cc:1939] T a43359647ee446a1b6cd04d6906bec71 P c0f8745082c44f37a47d562686cf2780: Deleting consensus metadata
W20260619 14:32:31.343598 9303 connection.cc:441] server connection from 127.0.0.1:49276 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49276 (request call id 1) could send its response
I20260619 14:32:31.346282 9427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:31.329713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37434 (local address 127.2.2.61:36895)
0619 14:32:31.329838 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:31.329843 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:31.335352 (+ 5509us) server_negotiation.cc:407] Connection header received
0619 14:32:31.335560 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:31.335564 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:31.335625 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:31.335686 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:31.337013 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:31.337777 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:31.339398 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:31.339766 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:31.339849 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:31.342350 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:31.342354 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:31.342357 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:31.342388 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:31.344262 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:31.344869 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:31.344873 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:31.344875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:31.344959 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:31.345252 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:31.345255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:31.345257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:31.345466 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:31.345575 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:31.345892 (+ 317us) server_negotiation.cc:299] Negotiation successful
0619 14:32:31.346013 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 14:32:31.353709 8901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:31.342918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37450 (local address 127.2.2.61:36895)
0619 14:32:31.343036 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:31.343040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:31.344023 (+ 983us) server_negotiation.cc:407] Connection header received
0619 14:32:31.344210 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:31.344213 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:31.344264 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:31.344318 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:31.345373 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:31.346280 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:31.347392 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:31.347730 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:31.347800 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:31.349704 (+ 1904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:31.349709 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:31.349713 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:31.349745 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:31.351781 (+ 2036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:31.352310 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:31.352317 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:31.352320 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:31.352397 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:31.352707 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:31.352714 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:31.352717 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:31.352941 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:31.353047 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:31.353401 (+ 354us) server_negotiation.cc:299] Negotiation successful
0619 14:32:31.353521 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 14:32:31.353940 9147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:31.344181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56034 (local address 127.2.2.60:45719)
0619 14:32:31.344302 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:31.344308 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:31.345107 (+ 799us) server_negotiation.cc:407] Connection header received
0619 14:32:31.345364 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:31.345367 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:31.345417 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:31.345467 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:31.346287 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:31.347089 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:31.348188 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:31.348555 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:31.348642 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:31.350398 (+ 1756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:31.350403 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:31.350409 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:31.350438 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:31.352157 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:31.352661 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:31.352667 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:31.352670 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:31.352753 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:31.353044 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:31.353047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:31.353049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:31.353243 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:31.353349 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:31.353680 (+ 331us) server_negotiation.cc:299] Negotiation successful
0619 14:32:31.353794 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 14:32:31.357246 9428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:31.330540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56032 (local address 127.2.2.60:45719)
0619 14:32:31.330655 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:31.330660 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:31.347828 (+ 17168us) server_negotiation.cc:407] Connection header received
0619 14:32:31.347974 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:31.347977 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:31.348025 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:31.348072 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:31.349437 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:31.350212 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:31.351335 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:31.351679 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:31.351748 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:31.353293 (+ 1545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:31.353297 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:31.353300 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:31.353325 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:31.355548 (+ 2223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:31.356010 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:31.356013 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:31.356014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:31.356126 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:31.356388 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:31.356390 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:31.356392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:31.356551 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:31.356652 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:31.356914 (+ 262us) server_negotiation.cc:299] Negotiation successful
0619 14:32:31.357046 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 14:32:32.319478 9484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:32.318951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:39059 (local address 127.2.2.1:42791)
0619 14:32:32.319280 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:32.319376 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:39059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":85,"threads_started":1}
I20260619 14:32:32.319762 9485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:32.319128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:39059 (local address 127.2.2.3:47203)
0619 14:32:32.319346 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:32.319417 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:39059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":68,"threads_started":1}
W20260619 14:32:32.319793 9143 heartbeater.cc:644] Failed to heartbeat to 127.2.2.62:39059 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.2.62:39059: connect: Connection refused (error 111)
W20260619 14:32:32.320066 9416 heartbeater.cc:644] Failed to heartbeat to 127.2.2.62:39059 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.2.62:39059: connect: Connection refused (error 111)
I20260619 14:32:32.320335 9486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:32.319828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:39059 (local address 127.2.2.2:39933)
0619 14:32:32.320143 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:32.320220 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:39059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":76,"threads_started":1}
W20260619 14:32:32.320569 9280 heartbeater.cc:644] Failed to heartbeat to 127.2.2.62:39059 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.2.62:39059: connect: Connection refused (error 111)
I20260619 14:32:32.825789 9487 raft_consensus.cc:482] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cf27644a9d194a898a1d93d6821e8b38)
I20260619 14:32:32.825975 9488 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cf27644a9d194a898a1d93d6821e8b38)
I20260619 14:32:32.826048 9487 raft_consensus.cc:504] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cf27644a9d194a898a1d93d6821e8b38" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 39059 } } peers { permanent_uuid: "e36baac4eca2403197084187e1360c08" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 36895 } attrs { promote: false } } peers { permanent_uuid: "03d127796b8d4c3286344b40cff82093" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 45719 } attrs { promote: false } }
I20260619 14:32:32.826159 9488 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cf27644a9d194a898a1d93d6821e8b38" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 39059 } } peers { permanent_uuid: "e36baac4eca2403197084187e1360c08" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 36895 } attrs { promote: false } } peers { permanent_uuid: "03d127796b8d4c3286344b40cff82093" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 45719 } attrs { promote: false } }
I20260619 14:32:32.826727 9488 leader_election.cc:288] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cf27644a9d194a898a1d93d6821e8b38 (127.2.2.62:39059), e36baac4eca2403197084187e1360c08 (127.2.2.61:36895)
I20260619 14:32:32.827394 9489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:32.826878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:39059 (local address 127.0.0.1:38262)
0619 14:32:32.827183 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:32.827282 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:39059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":71,"threads_started":1}
I20260619 14:32:32.827570 9487 leader_election.cc:288] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cf27644a9d194a898a1d93d6821e8b38 (127.2.2.62:39059), 03d127796b8d4c3286344b40cff82093 (127.2.2.60:45719)
W20260619 14:32:32.827586 8949 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.2.62:39059: connect: Connection refused (error 111)
I20260619 14:32:32.827865 9492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:32.827351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:39059 (local address 127.0.0.1:38264)
0619 14:32:32.827670 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:32.827777 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:39059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":60,"threads_started":1}
W20260619 14:32:32.828027 8764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.2.62:39059: connect: Connection refused (error 111)
I20260619 14:32:32.828593 9489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:32.828365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:39059 (local address 127.0.0.1:38278)
0619 14:32:32.828454 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:32.828504 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:39059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260619 14:32:32.828737 8949 leader_election.cc:334] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cf27644a9d194a898a1d93d6821e8b38 (127.2.2.62:39059): Network error: Client connection negotiation failed: client connection to 127.2.2.62:39059: connect: Connection refused (error 111)
I20260619 14:32:32.828904 9492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:32.828722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:39059 (local address 127.0.0.1:38288)
0619 14:32:32.828805 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:32.828857 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:39059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260619 14:32:32.829013 8764 leader_election.cc:334] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cf27644a9d194a898a1d93d6821e8b38 (127.2.2.62:39059): Network error: Client connection negotiation failed: client connection to 127.2.2.62:39059: connect: Connection refused (error 111)
I20260619 14:32:32.832198 9490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:32.826963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.61:36895 (local address 127.0.0.1:37454)
0619 14:32:32.827223 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:32.827239 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:32.827349 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:32.828362 (+ 1013us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:32.828367 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:32.828384 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:32:32.828502 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:32.828508 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:32.829634 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:32.829637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:32.830909 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:32.830919 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:32.831781 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:32.831784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:32.831851 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:32.831872 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 14:32:32.831939 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 14:32:32.832047 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":63,"threads_started":1}
I20260619 14:32:32.832180 9491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:32.827079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37454 (local address 127.2.2.61:36895)
0619 14:32:32.827812 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:32.827816 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:32.827831 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:32:32.827878 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:32.827881 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:32.828071 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:32.828606 (+ 535us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:32:32.828713 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:32.829525 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:32.831059 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:32.831658 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:32.831773 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:32.831835 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:32.831983 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 14:32:32.832078 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":65,"threads_started":1}
I20260619 14:32:32.832455 9493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:32.827380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.60:45719 (local address 127.0.0.1:56046)
0619 14:32:32.827853 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:32.827867 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:32.827956 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:32.828877 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:32.828880 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:32.828902 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:32:32.828977 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:32.828983 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:32.829933 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:32.829936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:32.831142 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:32.831150 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:32.832020 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:32.832023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:32.832120 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:32.832142 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 14:32:32.832243 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 14:32:32.832338 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":65,"threads_started":1}
I20260619 14:32:32.832460 9495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:32.827903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56046 (local address 127.2.2.60:45719)
0619 14:32:32.828175 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:32.828183 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:32.828195 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:32:32.828236 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:32.828239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:32.828293 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:32.829082 (+ 789us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:32:32.829169 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:32.829828 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:32.831255 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:32.831885 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:32.832006 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:32.832056 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:32.832251 (+ 195us) server_negotiation.cc:299] Negotiation successful
0619 14:32:32.832339 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":81,"threads_started":1}
I20260619 14:32:32.832930 8789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "03d127796b8d4c3286344b40cff82093" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e36baac4eca2403197084187e1360c08" is_pre_election: true
I20260619 14:32:32.832996 8973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e36baac4eca2403197084187e1360c08" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "03d127796b8d4c3286344b40cff82093" is_pre_election: true
I20260619 14:32:32.833122 8789 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03d127796b8d4c3286344b40cff82093 in term 1.
I20260619 14:32:32.833227 8973 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e36baac4eca2403197084187e1360c08 in term 1.
I20260619 14:32:32.833451 8948 leader_election.cc:302] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 03d127796b8d4c3286344b40cff82093, e36baac4eca2403197084187e1360c08; no voters: cf27644a9d194a898a1d93d6821e8b38
I20260619 14:32:32.833560 8766 leader_election.cc:302] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 03d127796b8d4c3286344b40cff82093, e36baac4eca2403197084187e1360c08; no voters: cf27644a9d194a898a1d93d6821e8b38
I20260619 14:32:32.833643 9488 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 14:32:32.833707 9488 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093 [term 1 FOLLOWER]: Starting leader election (detected failure of leader cf27644a9d194a898a1d93d6821e8b38)
I20260619 14:32:32.833760 9488 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093 [term 1 FOLLOWER]: Advancing to term 2
I20260619 14:32:32.833784 9487 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 14:32:32.833839 9487 raft_consensus.cc:482] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [term 1 FOLLOWER]: Starting leader election (detected failure of leader cf27644a9d194a898a1d93d6821e8b38)
I20260619 14:32:32.833884 9487 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [term 1 FOLLOWER]: Advancing to term 2
I20260619 14:32:32.834913 9487 raft_consensus.cc:504] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cf27644a9d194a898a1d93d6821e8b38" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 39059 } } peers { permanent_uuid: "e36baac4eca2403197084187e1360c08" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 36895 } attrs { promote: false } } peers { permanent_uuid: "03d127796b8d4c3286344b40cff82093" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 45719 } attrs { promote: false } }
I20260619 14:32:32.834914 9488 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cf27644a9d194a898a1d93d6821e8b38" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 39059 } } peers { permanent_uuid: "e36baac4eca2403197084187e1360c08" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 36895 } attrs { promote: false } } peers { permanent_uuid: "03d127796b8d4c3286344b40cff82093" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 45719 } attrs { promote: false } }
I20260619 14:32:32.835318 9488 leader_election.cc:288] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093 [CANDIDATE]: Term 2 election: Requested vote from peers cf27644a9d194a898a1d93d6821e8b38 (127.2.2.62:39059), e36baac4eca2403197084187e1360c08 (127.2.2.61:36895)
I20260619 14:32:32.835320 9487 leader_election.cc:288] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [CANDIDATE]: Term 2 election: Requested vote from peers cf27644a9d194a898a1d93d6821e8b38 (127.2.2.62:39059), 03d127796b8d4c3286344b40cff82093 (127.2.2.60:45719)
I20260619 14:32:32.835647 9490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:32.835414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:39059 (local address 127.0.0.1:38304)
0619 14:32:32.835520 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:32.835580 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:39059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260619 14:32:32.835708 8789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "03d127796b8d4c3286344b40cff82093" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e36baac4eca2403197084187e1360c08"
I20260619 14:32:32.835748 8973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e36baac4eca2403197084187e1360c08" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "03d127796b8d4c3286344b40cff82093"
I20260619 14:32:32.835841 8789 raft_consensus.cc:2379] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 03d127796b8d4c3286344b40cff82093 in current term 2: Already voted for candidate e36baac4eca2403197084187e1360c08 in this term.
I20260619 14:32:32.835911 8973 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate e36baac4eca2403197084187e1360c08 in current term 2: Already voted for candidate 03d127796b8d4c3286344b40cff82093 in this term.
I20260619 14:32:32.835917 9493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:32.835413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:39059 (local address 127.0.0.1:38302)
0619 14:32:32.835771 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:32.835823 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:39059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287}
I20260619 14:32:32.836544 9493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:32.836322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:39059 (local address 127.0.0.1:38316)
0619 14:32:32.836423 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:32.836484 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:39059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260619 14:32:32.836642 8764 leader_election.cc:334] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer cf27644a9d194a898a1d93d6821e8b38 (127.2.2.62:39059): Network error: Client connection negotiation failed: client connection to 127.2.2.62:39059: connect: Connection refused (error 111)
I20260619 14:32:32.836706 8764 leader_election.cc:302] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e36baac4eca2403197084187e1360c08; no voters: 03d127796b8d4c3286344b40cff82093, cf27644a9d194a898a1d93d6821e8b38
I20260619 14:32:32.836807 9490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:32.836550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:39059 (local address 127.0.0.1:38326)
0619 14:32:32.836684 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:32.836744 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:39059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 14:32:32.836833 9487 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260619 14:32:32.836943 8949 leader_election.cc:334] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer cf27644a9d194a898a1d93d6821e8b38 (127.2.2.62:39059): Network error: Client connection negotiation failed: client connection to 127.2.2.62:39059: connect: Connection refused (error 111)
I20260619 14:32:32.837010 8949 leader_election.cc:302] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03d127796b8d4c3286344b40cff82093; no voters: cf27644a9d194a898a1d93d6821e8b38, e36baac4eca2403197084187e1360c08
I20260619 14:32:32.837133 9488 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260619 14:32:33.321012 9497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:33.320495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:39059 (local address 127.2.2.1:48101)
0619 14:32:33.320823 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:33.320903 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:39059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":112,"threads_started":1}
I20260619 14:32:33.321427 9498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:33.320682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:39059 (local address 127.2.2.3:34291)
0619 14:32:33.321257 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:33.321325 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:39059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":75,"threads_started":1}
I20260619 14:32:33.321610 9499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:33.321116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:39059 (local address 127.2.2.2:44879)
0619 14:32:33.321351 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:33.321415 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:39059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":84,"threads_started":1}
I20260619 14:32:34.322450 9500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:34.321882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:39059 (local address 127.2.2.1:44567)
0619 14:32:34.322246 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:34.322322 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:39059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":91,"threads_started":1}
I20260619 14:32:34.322757 9501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:34.322150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:39059 (local address 127.2.2.3:60799)
0619 14:32:34.322422 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:34.322488 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:39059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":60,"threads_started":1}
I20260619 14:32:34.322793 9502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:34.322331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:39059 (local address 127.2.2.2:37319)
0619 14:32:34.322621 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:34.322705 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:39059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":62,"threads_started":1}
I20260619 14:32:34.877804 9503 raft_consensus.cc:482] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:32:34.877949 9503 raft_consensus.cc:504] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cf27644a9d194a898a1d93d6821e8b38" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 39059 } } peers { permanent_uuid: "e36baac4eca2403197084187e1360c08" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 36895 } attrs { promote: false } } peers { permanent_uuid: "03d127796b8d4c3286344b40cff82093" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 45719 } attrs { promote: false } }
I20260619 14:32:34.878506 9503 leader_election.cc:288] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cf27644a9d194a898a1d93d6821e8b38 (127.2.2.62:39059), 03d127796b8d4c3286344b40cff82093 (127.2.2.60:45719)
I20260619 14:32:34.879001 8973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e36baac4eca2403197084187e1360c08" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "03d127796b8d4c3286344b40cff82093" is_pre_election: true
I20260619 14:32:34.879161 9504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:34.878739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:39059 (local address 127.0.0.1:38586)
0619 14:32:34.879016 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:34.879074 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:39059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":61,"threads_started":1}
I20260619 14:32:34.879220 8973 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e36baac4eca2403197084187e1360c08 in term 2.
I20260619 14:32:34.879624 8766 leader_election.cc:302] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03d127796b8d4c3286344b40cff82093, e36baac4eca2403197084187e1360c08; no voters:
I20260619 14:32:34.879830 9503 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260619 14:32:34.879904 9503 raft_consensus.cc:482] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:32:34.879946 9503 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [term 2 FOLLOWER]: Advancing to term 3
I20260619 14:32:34.880227 9504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:34.879940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:39059 (local address 127.0.0.1:38590)
0619 14:32:34.880116 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:34.880169 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:39059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 14:32:34.880338 8764 leader_election.cc:334] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer cf27644a9d194a898a1d93d6821e8b38 (127.2.2.62:39059): Network error: Client connection negotiation failed: client connection to 127.2.2.62:39059: connect: Connection refused (error 111)
I20260619 14:32:34.881153 9503 raft_consensus.cc:504] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cf27644a9d194a898a1d93d6821e8b38" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 39059 } } peers { permanent_uuid: "e36baac4eca2403197084187e1360c08" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 36895 } attrs { promote: false } } peers { permanent_uuid: "03d127796b8d4c3286344b40cff82093" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 45719 } attrs { promote: false } }
I20260619 14:32:34.881493 9503 leader_election.cc:288] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [CANDIDATE]: Term 3 election: Requested vote from peers cf27644a9d194a898a1d93d6821e8b38 (127.2.2.62:39059), 03d127796b8d4c3286344b40cff82093 (127.2.2.60:45719)
I20260619 14:32:34.881902 8973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e36baac4eca2403197084187e1360c08" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "03d127796b8d4c3286344b40cff82093"
I20260619 14:32:34.882017 9504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:34.881808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:39059 (local address 127.0.0.1:38596)
0619 14:32:34.881919 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:34.881965 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:39059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 14:32:34.882082 8973 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093 [term 2 FOLLOWER]: Advancing to term 3
I20260619 14:32:34.882745 9504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:34.882534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:39059 (local address 127.0.0.1:38602)
0619 14:32:34.882638 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:34.882687 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:39059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 14:32:34.882843 8764 leader_election.cc:334] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer cf27644a9d194a898a1d93d6821e8b38 (127.2.2.62:39059): Network error: Client connection negotiation failed: client connection to 127.2.2.62:39059: connect: Connection refused (error 111)
I20260619 14:32:34.883239 8973 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e36baac4eca2403197084187e1360c08 in term 3.
I20260619 14:32:34.883579 8766 leader_election.cc:302] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 03d127796b8d4c3286344b40cff82093, e36baac4eca2403197084187e1360c08; no voters: cf27644a9d194a898a1d93d6821e8b38
I20260619 14:32:34.883788 9503 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 14:32:34.883983 9503 raft_consensus.cc:686] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [term 3 LEADER]: Becoming Leader. State: Replica: e36baac4eca2403197084187e1360c08, State: Running, Role: LEADER
I20260619 14:32:34.884294 9503 consensus_queue.cc:237] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cf27644a9d194a898a1d93d6821e8b38" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 39059 } } peers { permanent_uuid: "e36baac4eca2403197084187e1360c08" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 36895 } attrs { promote: false } } peers { permanent_uuid: "03d127796b8d4c3286344b40cff82093" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 45719 } attrs { promote: false } }
I20260619 14:32:34.885382 9506 sys_catalog.cc:455] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e36baac4eca2403197084187e1360c08. Latest consensus state: current_term: 3 leader_uuid: "e36baac4eca2403197084187e1360c08" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cf27644a9d194a898a1d93d6821e8b38" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 39059 } } peers { permanent_uuid: "e36baac4eca2403197084187e1360c08" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 36895 } attrs { promote: false } } peers { permanent_uuid: "03d127796b8d4c3286344b40cff82093" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 45719 } attrs { promote: false } } }
I20260619 14:32:34.885506 9506 sys_catalog.cc:458] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [sys.catalog]: This master's current role is: LEADER
I20260619 14:32:34.885828 9508 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:32:34.888033 9508 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:32:34.888429 9508 catalog_manager.cc:1259] Loaded cluster ID: 96382bd00e6e4b0792b1bfa035e796dc
I20260619 14:32:34.888474 9508 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:32:34.889024 9508 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:32:34.889408 9508 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08: Loaded TSK: 0
I20260619 14:32:34.890185 9508 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:32:34.901454 8778 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37450:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:32:34.904825 8778 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:32:34.908308 9504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:34.908029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:39059 (local address 127.0.0.1:38606)
0619 14:32:34.908170 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:34.908236 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:39059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 14:32:34.908396 8973 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093 [term 3 FOLLOWER]: Refusing update from remote peer e36baac4eca2403197084187e1360c08: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260619 14:32:34.909000 9504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:34.908776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:39059 (local address 127.0.0.1:38618)
0619 14:32:34.908878 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:34.908934 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:39059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 14:32:34.908964 9506 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03d127796b8d4c3286344b40cff82093" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 45719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260619 14:32:34.909147 8764 consensus_peers.cc:487] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 -> Peer cf27644a9d194a898a1d93d6821e8b38 (127.2.2.62:39059): Couldn't send request to peer cf27644a9d194a898a1d93d6821e8b38. Status: Network error: Client connection negotiation failed: client connection to 127.2.2.62:39059: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 14:32:34.910533 9510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e36baac4eca2403197084187e1360c08. Latest consensus state: current_term: 3 leader_uuid: "e36baac4eca2403197084187e1360c08" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cf27644a9d194a898a1d93d6821e8b38" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 39059 } } peers { permanent_uuid: "e36baac4eca2403197084187e1360c08" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 36895 } attrs { promote: false } } peers { permanent_uuid: "03d127796b8d4c3286344b40cff82093" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 45719 } attrs { promote: false } } }
I20260619 14:32:34.910687 9510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:32:34.911513 9509 mvcc.cc:205] Tried to move back new op lower bound from 7298578656899018752 to 7298578656808587264. Current Snapshot: MvccSnapshot[applied={T|T < 7298578656899018752}]
I20260619 14:32:34.911756 9503 sys_catalog.cc:455] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "e36baac4eca2403197084187e1360c08" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cf27644a9d194a898a1d93d6821e8b38" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 39059 } } peers { permanent_uuid: "e36baac4eca2403197084187e1360c08" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 36895 } attrs { promote: false } } peers { permanent_uuid: "03d127796b8d4c3286344b40cff82093" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 45719 } attrs { promote: false } } }
I20260619 14:32:34.911893 9503 sys_catalog.cc:458] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08 [sys.catalog]: This master's current role is: LEADER
I20260619 14:32:34.914274 9510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "e36baac4eca2403197084187e1360c08" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cf27644a9d194a898a1d93d6821e8b38" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 39059 } } peers { permanent_uuid: "e36baac4eca2403197084187e1360c08" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 36895 } attrs { promote: false } } peers { permanent_uuid: "03d127796b8d4c3286344b40cff82093" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 45719 } attrs { promote: false } } }
I20260619 14:32:34.914402 9510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03d127796b8d4c3286344b40cff82093 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:32:34.922830 9517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:34.918234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.3:41487 (local address 127.0.0.1:49288)
0619 14:32:34.918919 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:34.918936 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:34.919063 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:34.919344 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:34.919347 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:34.919362 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:32:34.919434 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:34.919440 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:34.920327 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:34.920331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:34.921441 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:34.921449 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:34.922359 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:34.922361 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:34.922473 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:34.922503 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 14:32:34.922581 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 14:32:34.922669 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"mutex_wait_us":29,"thread_start_us":100,"threads_started":1}
I20260619 14:32:34.922830 9504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:34.917890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.1:45703 (local address 127.0.0.1:46672)
0619 14:32:34.917990 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:34.918008 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:34.918101 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:34.919033 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:34.919037 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:34.919053 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:32:34.919193 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:34.919202 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:34.920199 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:34.920206 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:34.921472 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:34.921482 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:34.922428 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:34.922431 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:34.922555 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:34.922582 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 14:32:34.922647 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 14:32:34.922699 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":45}
I20260619 14:32:34.923285 9518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:34.918395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49288 (local address 127.2.2.3:41487)
0619 14:32:34.918921 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:34.918925 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:34.919008 (+ 83us) server_negotiation.cc:407] Connection header received
0619 14:32:34.919205 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:34.919207 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:34.919247 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:34.919314 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:32:34.919534 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:34.920206 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:34.921611 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:34.922088 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:34.922221 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:34.922286 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:34.923102 (+ 816us) server_negotiation.cc:299] Negotiation successful
0619 14:32:34.923187 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":70,"threads_started":1}
I20260619 14:32:34.923743 9516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:34.918074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46672 (local address 127.2.2.1:45703)
0619 14:32:34.918479 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:34.918486 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:34.918786 (+ 300us) server_negotiation.cc:407] Connection header received
0619 14:32:34.918845 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:34.918848 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:34.918901 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:34.919322 (+ 421us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:32:34.919438 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:34.920091 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:34.921623 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:34.922319 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:34.922426 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:34.922529 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:34.922677 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 14:32:34.922761 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":117,"threads_started":1}
I20260619 14:32:34.924012 9350 tablet_service.cc:1467] Processing CreateTablet for tablet d4886994b6f649dd8433d7718480a86e (DEFAULT_TABLE table=test-table [id=a1523a0231cd404a88a1f971ef97333d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:32:34.924147 9519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:34.918513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.2:34039 (local address 127.0.0.1:37488)
0619 14:32:34.918834 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:34.918848 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:34.918937 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:34.920566 (+ 1629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:34.920569 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:34.920588 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:32:34.920669 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:34.920675 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:34.921651 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:34.921654 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:34.922810 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:34.922819 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:34.923776 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:34.923782 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:34.923860 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:34.923882 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 14:32:34.923958 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 14:32:34.924022 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":7,"thread_start_us":112,"threads_started":1}
I20260619 14:32:34.924192 9520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:34.918616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37488 (local address 127.2.2.2:34039)
0619 14:32:34.920345 (+ 1729us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:34.920349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:34.920363 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:32:34.920411 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:34.920414 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:34.920456 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:34.920542 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:32:34.920792 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:34.921533 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:34.922945 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:34.923643 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:34.923763 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:34.923838 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:34.923981 (+ 143us) server_negotiation.cc:299] Negotiation successful
0619 14:32:34.924060 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1652,"thread_start_us":92,"threads_started":1}
I20260619 14:32:34.924206 9077 tablet_service.cc:1467] Processing CreateTablet for tablet d4886994b6f649dd8433d7718480a86e (DEFAULT_TABLE table=test-table [id=a1523a0231cd404a88a1f971ef97333d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:32:34.924340 9350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4886994b6f649dd8433d7718480a86e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:34.924485 9077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4886994b6f649dd8433d7718480a86e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:34.926369 9521 tablet_bootstrap.cc:492] T d4886994b6f649dd8433d7718480a86e P c0f8745082c44f37a47d562686cf2780: Bootstrap starting.
I20260619 14:32:34.926841 9214 tablet_service.cc:1467] Processing CreateTablet for tablet d4886994b6f649dd8433d7718480a86e (DEFAULT_TABLE table=test-table [id=a1523a0231cd404a88a1f971ef97333d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:32:34.927102 9214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4886994b6f649dd8433d7718480a86e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:34.927366 9521 tablet_bootstrap.cc:654] T d4886994b6f649dd8433d7718480a86e P c0f8745082c44f37a47d562686cf2780: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:34.928593 9521 tablet_bootstrap.cc:492] T d4886994b6f649dd8433d7718480a86e P c0f8745082c44f37a47d562686cf2780: No bootstrap required, opened a new log
I20260619 14:32:34.928690 9521 ts_tablet_manager.cc:1397] T d4886994b6f649dd8433d7718480a86e P c0f8745082c44f37a47d562686cf2780: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 14:32:34.929127 9521 raft_consensus.cc:348] T d4886994b6f649dd8433d7718480a86e P c0f8745082c44f37a47d562686cf2780 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c81bd9d0e5f44e0fa56ac4dcc2401a27" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 45703 } } peers { permanent_uuid: "c0f8745082c44f37a47d562686cf2780" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 41487 } } peers { permanent_uuid: "3cc7f187edfc4edabbb95a0f1bdce56a" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 34039 } }
I20260619 14:32:34.929289 9521 raft_consensus.cc:374] T d4886994b6f649dd8433d7718480a86e P c0f8745082c44f37a47d562686cf2780 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:34.929325 9521 raft_consensus.cc:729] T d4886994b6f649dd8433d7718480a86e P c0f8745082c44f37a47d562686cf2780 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f8745082c44f37a47d562686cf2780, State: Initialized, Role: FOLLOWER
I20260619 14:32:34.929448 9521 consensus_queue.cc:260] T d4886994b6f649dd8433d7718480a86e P c0f8745082c44f37a47d562686cf2780 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c81bd9d0e5f44e0fa56ac4dcc2401a27" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 45703 } } peers { permanent_uuid: "c0f8745082c44f37a47d562686cf2780" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 41487 } } peers { permanent_uuid: "3cc7f187edfc4edabbb95a0f1bdce56a" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 34039 } }
I20260619 14:32:34.929826 9521 ts_tablet_manager.cc:1428] T d4886994b6f649dd8433d7718480a86e P c0f8745082c44f37a47d562686cf2780: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260619 14:32:34.929940 9524 tablet_bootstrap.cc:492] T d4886994b6f649dd8433d7718480a86e P 3cc7f187edfc4edabbb95a0f1bdce56a: Bootstrap starting.
I20260619 14:32:34.930209 9522 tablet_bootstrap.cc:492] T d4886994b6f649dd8433d7718480a86e P c81bd9d0e5f44e0fa56ac4dcc2401a27: Bootstrap starting.
I20260619 14:32:34.930721 9526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:34.930289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:39059 (local address 127.2.2.3:34233)
0619 14:32:34.930543 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:34.930612 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:39059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":92,"threads_started":1}
I20260619 14:32:34.931248 9522 tablet_bootstrap.cc:654] T d4886994b6f649dd8433d7718480a86e P c81bd9d0e5f44e0fa56ac4dcc2401a27: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:34.931262 9524 tablet_bootstrap.cc:654] T d4886994b6f649dd8433d7718480a86e P 3cc7f187edfc4edabbb95a0f1bdce56a: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:34.932322 9526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:34.932042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:39059 (local address 127.2.2.3:34197)
0619 14:32:34.932198 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:34.932256 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:39059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260619 14:32:34.932859 9524 tablet_bootstrap.cc:492] T d4886994b6f649dd8433d7718480a86e P 3cc7f187edfc4edabbb95a0f1bdce56a: No bootstrap required, opened a new log
I20260619 14:32:34.932942 9524 ts_tablet_manager.cc:1397] T d4886994b6f649dd8433d7718480a86e P 3cc7f187edfc4edabbb95a0f1bdce56a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 14:32:34.933229 9522 tablet_bootstrap.cc:492] T d4886994b6f649dd8433d7718480a86e P c81bd9d0e5f44e0fa56ac4dcc2401a27: No bootstrap required, opened a new log
I20260619 14:32:34.933307 9522 ts_tablet_manager.cc:1397] T d4886994b6f649dd8433d7718480a86e P c81bd9d0e5f44e0fa56ac4dcc2401a27: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 14:32:34.933373 9524 raft_consensus.cc:348] T d4886994b6f649dd8433d7718480a86e P 3cc7f187edfc4edabbb95a0f1bdce56a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c81bd9d0e5f44e0fa56ac4dcc2401a27" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 45703 } } peers { permanent_uuid: "c0f8745082c44f37a47d562686cf2780" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 41487 } } peers { permanent_uuid: "3cc7f187edfc4edabbb95a0f1bdce56a" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 34039 } }
I20260619 14:32:34.933485 9524 raft_consensus.cc:374] T d4886994b6f649dd8433d7718480a86e P 3cc7f187edfc4edabbb95a0f1bdce56a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:34.933526 9524 raft_consensus.cc:729] T d4886994b6f649dd8433d7718480a86e P 3cc7f187edfc4edabbb95a0f1bdce56a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cc7f187edfc4edabbb95a0f1bdce56a, State: Initialized, Role: FOLLOWER
I20260619 14:32:34.933657 9524 consensus_queue.cc:260] T d4886994b6f649dd8433d7718480a86e P 3cc7f187edfc4edabbb95a0f1bdce56a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c81bd9d0e5f44e0fa56ac4dcc2401a27" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 45703 } } peers { permanent_uuid: "c0f8745082c44f37a47d562686cf2780" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 41487 } } peers { permanent_uuid: "3cc7f187edfc4edabbb95a0f1bdce56a" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 34039 } }
I20260619 14:32:34.933611 9522 raft_consensus.cc:348] T d4886994b6f649dd8433d7718480a86e P c81bd9d0e5f44e0fa56ac4dcc2401a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c81bd9d0e5f44e0fa56ac4dcc2401a27" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 45703 } } peers { permanent_uuid: "c0f8745082c44f37a47d562686cf2780" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 41487 } } peers { permanent_uuid: "3cc7f187edfc4edabbb95a0f1bdce56a" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 34039 } }
I20260619 14:32:34.933750 9522 raft_consensus.cc:374] T d4886994b6f649dd8433d7718480a86e P c81bd9d0e5f44e0fa56ac4dcc2401a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:34.933777 9522 raft_consensus.cc:729] T d4886994b6f649dd8433d7718480a86e P c81bd9d0e5f44e0fa56ac4dcc2401a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c81bd9d0e5f44e0fa56ac4dcc2401a27, State: Initialized, Role: FOLLOWER
I20260619 14:32:34.933933 9522 consensus_queue.cc:260] T d4886994b6f649dd8433d7718480a86e P c81bd9d0e5f44e0fa56ac4dcc2401a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c81bd9d0e5f44e0fa56ac4dcc2401a27" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 45703 } } peers { permanent_uuid: "c0f8745082c44f37a47d562686cf2780" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 41487 } } peers { permanent_uuid: "3cc7f187edfc4edabbb95a0f1bdce56a" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 34039 } }
I20260619 14:32:34.933983 9524 ts_tablet_manager.cc:1428] T d4886994b6f649dd8433d7718480a86e P 3cc7f187edfc4edabbb95a0f1bdce56a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:32:34.934726 9418 heartbeater.cc:503] Master 127.2.2.61:36895 was elected leader, sending a full tablet report...
I20260619 14:32:34.935274 9522 ts_tablet_manager.cc:1428] T d4886994b6f649dd8433d7718480a86e P c81bd9d0e5f44e0fa56ac4dcc2401a27: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260619 14:32:34.934684 9530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:34.934242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:39059 (local address 127.2.2.2:35861)
0619 14:32:34.934495 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:34.934557 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:39059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":95,"threads_started":1}
I20260619 14:32:34.936776 9282 heartbeater.cc:503] Master 127.2.2.61:36895 was elected leader, sending a full tablet report...
I20260619 14:32:34.937544 9530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:34.937014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:39059 (local address 127.2.2.2:48357)
0619 14:32:34.937405 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:34.937463 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:39059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":101,"threads_started":1}
I20260619 14:32:34.938021 9532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:34.935003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:39059 (local address 127.2.2.1:56531)
0619 14:32:34.937849 (+ 2846us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:34.937922 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:39059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2789,"thread_start_us":91,"threads_started":1}
I20260619 14:32:34.938215 9145 heartbeater.cc:503] Master 127.2.2.61:36895 was elected leader, sending a full tablet report...
I20260619 14:32:34.939045 9532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:34.938723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:39059 (local address 127.2.2.1:36593)
0619 14:32:34.938956 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:34.938993 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:39059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":86,"threads_started":1}
W20260619 14:32:35.080053 9419 tablet.cc:2369] T d4886994b6f649dd8433d7718480a86e P c0f8745082c44f37a47d562686cf2780: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:32:35.115409 9283 tablet.cc:2369] T d4886994b6f649dd8433d7718480a86e P 3cc7f187edfc4edabbb95a0f1bdce56a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:32:35.140923 9146 tablet.cc:2369] T d4886994b6f649dd8433d7718480a86e P c81bd9d0e5f44e0fa56ac4dcc2401a27: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:32:35.174997 9531 raft_consensus.cc:482] T d4886994b6f649dd8433d7718480a86e P c81bd9d0e5f44e0fa56ac4dcc2401a27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:32:35.175146 9531 raft_consensus.cc:504] T d4886994b6f649dd8433d7718480a86e P c81bd9d0e5f44e0fa56ac4dcc2401a27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c81bd9d0e5f44e0fa56ac4dcc2401a27" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 45703 } } peers { permanent_uuid: "c0f8745082c44f37a47d562686cf2780" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 41487 } } peers { permanent_uuid: "3cc7f187edfc4edabbb95a0f1bdce56a" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 34039 } }
I20260619 14:32:35.175591 9531 leader_election.cc:288] T d4886994b6f649dd8433d7718480a86e P c81bd9d0e5f44e0fa56ac4dcc2401a27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0f8745082c44f37a47d562686cf2780 (127.2.2.3:41487), 3cc7f187edfc4edabbb95a0f1bdce56a (127.2.2.2:34039)
I20260619 14:32:35.176151 9233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4886994b6f649dd8433d7718480a86e" candidate_uuid: "c81bd9d0e5f44e0fa56ac4dcc2401a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cc7f187edfc4edabbb95a0f1bdce56a" is_pre_election: true
I20260619 14:32:35.176153 9370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4886994b6f649dd8433d7718480a86e" candidate_uuid: "c81bd9d0e5f44e0fa56ac4dcc2401a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f8745082c44f37a47d562686cf2780" is_pre_election: true
I20260619 14:32:35.176290 9233 raft_consensus.cc:2454] T d4886994b6f649dd8433d7718480a86e P 3cc7f187edfc4edabbb95a0f1bdce56a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c81bd9d0e5f44e0fa56ac4dcc2401a27 in term 0.
I20260619 14:32:35.176328 9370 raft_consensus.cc:2454] T d4886994b6f649dd8433d7718480a86e P c0f8745082c44f37a47d562686cf2780 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c81bd9d0e5f44e0fa56ac4dcc2401a27 in term 0.
I20260619 14:32:35.176682 9030 leader_election.cc:302] T d4886994b6f649dd8433d7718480a86e P c81bd9d0e5f44e0fa56ac4dcc2401a27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cc7f187edfc4edabbb95a0f1bdce56a, c81bd9d0e5f44e0fa56ac4dcc2401a27; no voters:
I20260619 14:32:35.176900 9531 raft_consensus.cc:2781] T d4886994b6f649dd8433d7718480a86e P c81bd9d0e5f44e0fa56ac4dcc2401a27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:32:35.176975 9531 raft_consensus.cc:482] T d4886994b6f649dd8433d7718480a86e P c81bd9d0e5f44e0fa56ac4dcc2401a27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:32:35.177031 9531 raft_consensus.cc:3037] T d4886994b6f649dd8433d7718480a86e P c81bd9d0e5f44e0fa56ac4dcc2401a27 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:35.178872 9531 raft_consensus.cc:504] T d4886994b6f649dd8433d7718480a86e P c81bd9d0e5f44e0fa56ac4dcc2401a27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c81bd9d0e5f44e0fa56ac4dcc2401a27" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 45703 } } peers { permanent_uuid: "c0f8745082c44f37a47d562686cf2780" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 41487 } } peers { permanent_uuid: "3cc7f187edfc4edabbb95a0f1bdce56a" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 34039 } }
I20260619 14:32:35.179260 9531 leader_election.cc:288] T d4886994b6f649dd8433d7718480a86e P c81bd9d0e5f44e0fa56ac4dcc2401a27 [CANDIDATE]: Term 1 election: Requested vote from peers c0f8745082c44f37a47d562686cf2780 (127.2.2.3:41487), 3cc7f187edfc4edabbb95a0f1bdce56a (127.2.2.2:34039)
I20260619 14:32:35.179690 9370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4886994b6f649dd8433d7718480a86e" candidate_uuid: "c81bd9d0e5f44e0fa56ac4dcc2401a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f8745082c44f37a47d562686cf2780"
I20260619 14:32:35.179689 9233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4886994b6f649dd8433d7718480a86e" candidate_uuid: "c81bd9d0e5f44e0fa56ac4dcc2401a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cc7f187edfc4edabbb95a0f1bdce56a"
I20260619 14:32:35.179807 9233 raft_consensus.cc:3037] T d4886994b6f649dd8433d7718480a86e P 3cc7f187edfc4edabbb95a0f1bdce56a [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:35.179836 9370 raft_consensus.cc:3037] T d4886994b6f649dd8433d7718480a86e P c0f8745082c44f37a47d562686cf2780 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:35.180763 9370 raft_consensus.cc:2454] T d4886994b6f649dd8433d7718480a86e P c0f8745082c44f37a47d562686cf2780 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c81bd9d0e5f44e0fa56ac4dcc2401a27 in term 1.
I20260619 14:32:35.180763 9233 raft_consensus.cc:2454] T d4886994b6f649dd8433d7718480a86e P 3cc7f187edfc4edabbb95a0f1bdce56a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c81bd9d0e5f44e0fa56ac4dcc2401a27 in term 1.
I20260619 14:32:35.181099 9032 leader_election.cc:302] T d4886994b6f649dd8433d7718480a86e P c81bd9d0e5f44e0fa56ac4dcc2401a27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0f8745082c44f37a47d562686cf2780, c81bd9d0e5f44e0fa56ac4dcc2401a27; no voters:
I20260619 14:32:35.181334 9531 raft_consensus.cc:2781] T d4886994b6f649dd8433d7718480a86e P c81bd9d0e5f44e0fa56ac4dcc2401a27 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:32:35.181540 9531 raft_consensus.cc:686] T d4886994b6f649dd8433d7718480a86e P c81bd9d0e5f44e0fa56ac4dcc2401a27 [term 1 LEADER]: Becoming Leader. State: Replica: c81bd9d0e5f44e0fa56ac4dcc2401a27, State: Running, Role: LEADER
I20260619 14:32:35.181701 9531 consensus_queue.cc:237] T d4886994b6f649dd8433d7718480a86e P c81bd9d0e5f44e0fa56ac4dcc2401a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c81bd9d0e5f44e0fa56ac4dcc2401a27" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 45703 } } peers { permanent_uuid: "c0f8745082c44f37a47d562686cf2780" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 41487 } } peers { permanent_uuid: "3cc7f187edfc4edabbb95a0f1bdce56a" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 34039 } }
I20260619 14:32:35.182531 9532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:35.182245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:39059 (local address 127.2.2.1:33675)
0619 14:32:35.182371 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:35.182438 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:39059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260619 14:32:35.183427 8779 catalog_manager.cc:5495] T d4886994b6f649dd8433d7718480a86e P c81bd9d0e5f44e0fa56ac4dcc2401a27 reported cstate change: term changed from 0 to 1, leader changed from <none> to c81bd9d0e5f44e0fa56ac4dcc2401a27 (127.2.2.1). New cstate: current_term: 1 leader_uuid: "c81bd9d0e5f44e0fa56ac4dcc2401a27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c81bd9d0e5f44e0fa56ac4dcc2401a27" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 45703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f8745082c44f37a47d562686cf2780" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 41487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cc7f187edfc4edabbb95a0f1bdce56a" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 34039 } health_report { overall_health: UNKNOWN } } }
I20260619 14:32:35.219515 9516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:35.216366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46676 (local address 127.2.2.1:45703)
0619 14:32:35.216525 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:35.216529 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:35.216543 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:32:35.216644 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:35.216647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:35.216711 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:35.216784 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:32:35.217001 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:35.217492 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:35.218528 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:35.218769 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:35.218847 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:35.219050 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:32:35.219118 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:32:35.219171 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:35.219286 (+ 115us) server_negotiation.cc:299] Negotiation successful
0619 14:32:35.219344 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 14:32:35.221613 9233 raft_consensus.cc:1261] T d4886994b6f649dd8433d7718480a86e P 3cc7f187edfc4edabbb95a0f1bdce56a [term 1 FOLLOWER]: Refusing update from remote peer c81bd9d0e5f44e0fa56ac4dcc2401a27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:32:35.221642 9370 raft_consensus.cc:1261] T d4886994b6f649dd8433d7718480a86e P c0f8745082c44f37a47d562686cf2780 [term 1 FOLLOWER]: Refusing update from remote peer c81bd9d0e5f44e0fa56ac4dcc2401a27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:32:35.222074 9531 consensus_queue.cc:1035] T d4886994b6f649dd8433d7718480a86e P c81bd9d0e5f44e0fa56ac4dcc2401a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cc7f187edfc4edabbb95a0f1bdce56a" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 34039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:32:35.222249 9535 consensus_queue.cc:1035] T d4886994b6f649dd8433d7718480a86e P c81bd9d0e5f44e0fa56ac4dcc2401a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f8745082c44f37a47d562686cf2780" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 41487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:32:35.223829 9526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:35.223522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:39059 (local address 127.2.2.3:42367)
0619 14:32:35.223682 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:35.223754 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:39059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260619 14:32:35.225986 9538 mvcc.cc:205] Tried to move back new op lower bound from 7298578658183507968 to 7298578658025431040. Current Snapshot: MvccSnapshot[applied={T|T < 7298578658183507968}]
I20260619 14:32:35.228034 9532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:35.227790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:39059 (local address 127.2.2.1:43257)
0619 14:32:35.227892 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:35.227952 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:39059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260619 14:32:35.229455 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:35.223980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:39059 (local address 127.2.2.2:49021)
0619 14:32:35.229304 (+ 5324us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:35.229368 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:39059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5230}
I20260619 14:32:35.229540 9526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:35.229207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:39059 (local address 127.2.2.3:43639)
0619 14:32:35.229407 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:35.229464 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:39059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260619 14:32:35.230247 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:35.229948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:39059 (local address 127.2.2.2:56989)
0619 14:32:35.230088 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:35.230163 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:39059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260619 14:32:35.234325 8779 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:32:35.234469 8779 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:32:35.238092 8779 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e36baac4eca2403197084187e1360c08: Sending DeleteTablet for 3 replicas of tablet d4886994b6f649dd8433d7718480a86e
I20260619 14:32:35.238773 9077 tablet_service.cc:1514] Processing DeleteTablet for tablet d4886994b6f649dd8433d7718480a86e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:32:35 UTC) from {username='oryx', principal='oryx/127.2.2.61@KRBTEST.COM'} at 127.0.0.1:46672
I20260619 14:32:35.238839 9350 tablet_service.cc:1514] Processing DeleteTablet for tablet d4886994b6f649dd8433d7718480a86e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:32:35 UTC) from {username='oryx', principal='oryx/127.2.2.61@KRBTEST.COM'} at 127.0.0.1:49288
I20260619 14:32:35.239001 9214 tablet_service.cc:1514] Processing DeleteTablet for tablet d4886994b6f649dd8433d7718480a86e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:32:35 UTC) from {username='oryx', principal='oryx/127.2.2.61@KRBTEST.COM'} at 127.0.0.1:37488
I20260619 14:32:35.239136 9548 tablet_replica.cc:330] T d4886994b6f649dd8433d7718480a86e P c0f8745082c44f37a47d562686cf2780: stopping tablet replica
I20260619 14:32:35.239252 9548 raft_consensus.cc:2229] T d4886994b6f649dd8433d7718480a86e P c0f8745082c44f37a47d562686cf2780 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:32:35.239326 9547 tablet_replica.cc:330] T d4886994b6f649dd8433d7718480a86e P c81bd9d0e5f44e0fa56ac4dcc2401a27: stopping tablet replica
I20260619 14:32:35.239530 9549 tablet_replica.cc:330] T d4886994b6f649dd8433d7718480a86e P 3cc7f187edfc4edabbb95a0f1bdce56a: stopping tablet replica
I20260619 14:32:35.239553 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 8924
I20260619 14:32:35.239677 9549 raft_consensus.cc:2229] T d4886994b6f649dd8433d7718480a86e P 3cc7f187edfc4edabbb95a0f1bdce56a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:32:35.239394 9548 raft_consensus.cc:2258] T d4886994b6f649dd8433d7718480a86e P c0f8745082c44f37a47d562686cf2780 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:32:35.240828 9548 ts_tablet_manager.cc:1905] T d4886994b6f649dd8433d7718480a86e P c0f8745082c44f37a47d562686cf2780: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:32:35.239825 9549 raft_consensus.cc:2258] T d4886994b6f649dd8433d7718480a86e P 3cc7f187edfc4edabbb95a0f1bdce56a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:32:35.241701 9549 ts_tablet_manager.cc:1905] T d4886994b6f649dd8433d7718480a86e P 3cc7f187edfc4edabbb95a0f1bdce56a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:32:35.243100 9548 ts_tablet_manager.cc:1918] T d4886994b6f649dd8433d7718480a86e P c0f8745082c44f37a47d562686cf2780: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:32:35.243201 9548 log.cc:1196] T d4886994b6f649dd8433d7718480a86e P c0f8745082c44f37a47d562686cf2780: Deleting WAL directory at /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-2/wal/wals/d4886994b6f649dd8433d7718480a86e
I20260619 14:32:35.243458 9548 ts_tablet_manager.cc:1939] T d4886994b6f649dd8433d7718480a86e P c0f8745082c44f37a47d562686cf2780: Deleting consensus metadata
I20260619 14:32:35.244163 8765 catalog_manager.cc:4841] TS c0f8745082c44f37a47d562686cf2780 (127.2.2.3:41487): tablet d4886994b6f649dd8433d7718480a86e (table test-table [id=a1523a0231cd404a88a1f971ef97333d]) successfully deleted
I20260619 14:32:35.244976 9549 ts_tablet_manager.cc:1918] T d4886994b6f649dd8433d7718480a86e P 3cc7f187edfc4edabbb95a0f1bdce56a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:32:35.245065 9549 log.cc:1196] T d4886994b6f649dd8433d7718480a86e P 3cc7f187edfc4edabbb95a0f1bdce56a: Deleting WAL directory at /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879514993456-2056-0/minicluster-data/ts-1/wal/wals/d4886994b6f649dd8433d7718480a86e
I20260619 14:32:35.245337 9549 ts_tablet_manager.cc:1939] T d4886994b6f649dd8433d7718480a86e P 3cc7f187edfc4edabbb95a0f1bdce56a: Deleting consensus metadata
I20260619 14:32:35.246294 8763 catalog_manager.cc:4841] TS 3cc7f187edfc4edabbb95a0f1bdce56a (127.2.2.2:34039): tablet d4886994b6f649dd8433d7718480a86e (table test-table [id=a1523a0231cd404a88a1f971ef97333d]) successfully deleted
I20260619 14:32:35.249004 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 9151
I20260619 14:32:35.249353 9519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:35.249032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.1:45703 (local address 127.0.0.1:46686)
0619 14:32:35.249186 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:35.249251 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.1:45703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 14:32:35.249734 8764 catalog_manager.cc:4584] TS c81bd9d0e5f44e0fa56ac4dcc2401a27 (127.2.2.1:45703): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d4886994b6f649dd8433d7718480a86e: Network error: Client connection negotiation failed: client connection to 127.2.2.1:45703: connect: Connection refused (error 111)
I20260619 14:32:35.255007 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 9287
I20260619 14:32:35.260869 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 8642
I20260619 14:32:35.266732 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 8743
2026-06-19T14:32:35Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7624 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:32:35 dist-test-slave-70rm krb5kdc[9552](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:32:35 dist-test-slave-70rm krb5kdc[9552](info): set up 2 sockets
Jun 19 14:32:35 dist-test-slave-70rm krb5kdc[9552](info): commencing operation
krb5kdc: starting...
W20260619 14:32:36.310801 2056 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Jun 19 14:32:36 dist-test-slave-70rm krb5kdc[9552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:32:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:32:36Z Disabled control of system clock
WARNING: no policy specified for oryx/127.2.2.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.2.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.2.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/krb5kdc/oryx_127.2.2.62.keytab.
Entry for principal oryx/127.2.2.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/krb5kdc/oryx_127.2.2.62.keytab.
I20260619 14:32:36.434700 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:38669
--webserver_interface=127.2.2.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/krb5kdc/oryx_127.2.2.62.keytab
--principal=oryx/127.2.2.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:36765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.2.62:38669
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:32:36.545166 9565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:36.545457 9565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:36.545500 9565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:36.549665 9565 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:32:36.549739 9565 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 14:32:36.549757 9565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:36.549777 9565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:36.549793 9565 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:32:36.549852 9565 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:32:36.556885 9565 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:36765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.2.62:38669
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/krb5kdc/oryx_127.2.2.62.keytab
--principal=oryx/127.2.2.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:38669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.2.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9565
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:32:36.558043 9565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:36.559144 9565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:32:36.568081 9571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:36.568131 9565 server_base.cc:1034] running on GCE node
W20260619 14:32:36.568186 9570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:36.568156 9573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:36.568974 9565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:36.570115 9565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:36.571355 9565 hybrid_clock.cc:648] HybridClock initialized: now 1781879556571340 us; error 37 us; skew 500 ppm
Jun 19 14:32:36 dist-test-slave-70rm krb5kdc[9552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879556, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.2.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:32:36.574235 9565 init.cc:377] Logged in from keytab as oryx/127.2.2.62@KRBTEST.COM (short username oryx)
I20260619 14:32:36.575181 9565 webserver.cc:460] Webserver started at http://127.2.2.62:45085/ using document root <none> and password file <none>
I20260619 14:32:36.575759 9565 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:36.575819 9565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:36.576057 9565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:32:36.578305 9565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/master-0/data/instance:
uuid: "50728c18eca74389b5c824d5d291fa09"
format_stamp: "Formatted at 2026-06-19 14:32:36 on dist-test-slave-70rm"
server_key: "27c7cdc733e4c0747ee57b7567241427"
server_key_iv: "c04c6d4de54195ed9930445d881ac4d3"
server_key_version: "encryptionkey@0"
I20260619 14:32:36.578895 9565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/master-0/wal/instance:
uuid: "50728c18eca74389b5c824d5d291fa09"
format_stamp: "Formatted at 2026-06-19 14:32:36 on dist-test-slave-70rm"
server_key: "27c7cdc733e4c0747ee57b7567241427"
server_key_iv: "c04c6d4de54195ed9930445d881ac4d3"
server_key_version: "encryptionkey@0"
I20260619 14:32:36.582806 9565 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260619 14:32:36.585348 9580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:36.586699 9565 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:32:36.586843 9565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/master-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/master-0/wal
uuid: "50728c18eca74389b5c824d5d291fa09"
format_stamp: "Formatted at 2026-06-19 14:32:36 on dist-test-slave-70rm"
server_key: "27c7cdc733e4c0747ee57b7567241427"
server_key_iv: "c04c6d4de54195ed9930445d881ac4d3"
server_key_version: "encryptionkey@0"
I20260619 14:32:36.586957 9565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:36.609090 9565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:36.612789 9565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:32:36.613052 9565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:36.621948 9565 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.62:38669
I20260619 14:32:36.621964 9642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.62:38669 every 8 connection(s)
I20260619 14:32:36.623045 9565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
I20260619 14:32:36.625947 9643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:36.631321 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 9565
I20260619 14:32:36.631453 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/master-0/wal/instance
I20260619 14:32:36.631780 2056 external_mini_cluster.cc:1428] Setting key 0dede7ed19ceea5e54cf515f4d0e3e0d
I20260619 14:32:36.631904 9643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50728c18eca74389b5c824d5d291fa09: Bootstrap starting.
I20260619 14:32:36.634524 9643 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50728c18eca74389b5c824d5d291fa09: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:36.635270 9643 log.cc:824] T 00000000000000000000000000000000 P 50728c18eca74389b5c824d5d291fa09: Log is configured to *not* fsync() on all Append() calls
I20260619 14:32:36.637188 9643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50728c18eca74389b5c824d5d291fa09: No bootstrap required, opened a new log
Jun 19 14:32:36 dist-test-slave-70rm krb5kdc[9552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM
I20260619 14:32:36.640959 9643 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50728c18eca74389b5c824d5d291fa09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50728c18eca74389b5c824d5d291fa09" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 38669 } }
I20260619 14:32:36.641237 9643 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50728c18eca74389b5c824d5d291fa09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:36.641315 9643 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50728c18eca74389b5c824d5d291fa09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50728c18eca74389b5c824d5d291fa09, State: Initialized, Role: FOLLOWER
I20260619 14:32:36.641804 9643 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50728c18eca74389b5c824d5d291fa09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50728c18eca74389b5c824d5d291fa09" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 38669 } }
I20260619 14:32:36.641963 9643 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50728c18eca74389b5c824d5d291fa09 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:32:36.642050 9643 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50728c18eca74389b5c824d5d291fa09 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:32:36.642177 9643 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50728c18eca74389b5c824d5d291fa09 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:36.643110 9643 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50728c18eca74389b5c824d5d291fa09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50728c18eca74389b5c824d5d291fa09" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 38669 } }
I20260619 14:32:36.643535 9643 leader_election.cc:302] T 00000000000000000000000000000000 P 50728c18eca74389b5c824d5d291fa09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50728c18eca74389b5c824d5d291fa09; no voters:
I20260619 14:32:36.643929 9643 leader_election.cc:288] T 00000000000000000000000000000000 P 50728c18eca74389b5c824d5d291fa09 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:32:36.644097 9648 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50728c18eca74389b5c824d5d291fa09 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:32:36.644341 9648 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50728c18eca74389b5c824d5d291fa09 [term 1 LEADER]: Becoming Leader. State: Replica: 50728c18eca74389b5c824d5d291fa09, State: Running, Role: LEADER
I20260619 14:32:36.644708 9648 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50728c18eca74389b5c824d5d291fa09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50728c18eca74389b5c824d5d291fa09" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 38669 } }
I20260619 14:32:36.645401 9643 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50728c18eca74389b5c824d5d291fa09 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:32:36.646551 9650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50728c18eca74389b5c824d5d291fa09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50728c18eca74389b5c824d5d291fa09. Latest consensus state: current_term: 1 leader_uuid: "50728c18eca74389b5c824d5d291fa09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50728c18eca74389b5c824d5d291fa09" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 38669 } } }
I20260619 14:32:36.646677 9650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50728c18eca74389b5c824d5d291fa09 [sys.catalog]: This master's current role is: LEADER
I20260619 14:32:36.646847 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:36.633222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33070 (local address 127.2.2.62:38669)
0619 14:32:36.633713 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:36.633729 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:36.633773 (+ 44us) server_negotiation.cc:407] Connection header received
0619 14:32:36.634526 (+ 753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:36.634545 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:36.634941 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:36.635289 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:36.636109 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:36.637081 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:36.638012 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:36.638567 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:36.638743 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:36.641248 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:36.641256 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:36.641275 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:36.641308 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:36.643228 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:36.643822 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:36.643831 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:36.643838 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:36.643994 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:36.644329 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:36.644333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:36.644335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:36.644873 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:36.645053 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:36.645393 (+ 340us) server_negotiation.cc:299] Negotiation successful
0619 14:32:36.645707 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":158,"threads_started":1}
I20260619 14:32:36.646986 9649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50728c18eca74389b5c824d5d291fa09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50728c18eca74389b5c824d5d291fa09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50728c18eca74389b5c824d5d291fa09" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 38669 } } }
I20260619 14:32:36.647076 9649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50728c18eca74389b5c824d5d291fa09 [sys.catalog]: This master's current role is: LEADER
W20260619 14:32:36.650707 9663 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 50728c18eca74389b5c824d5d291fa09: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:32:36.650803 9663 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:32:36.650883 9657 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:32:36.651605 9657 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:32:36.657426 9657 catalog_manager.cc:1347] Generated new cluster ID: 12d9c988417a4f6ab27613f34684dee7
I20260619 14:32:36.657500 9657 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:32:36.662595 9657 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:32:36.663501 9657 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:32:36.668552 9657 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50728c18eca74389b5c824d5d291fa09: Generated new TSK 0
I20260619 14:32:36.669462 9657 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.2.2.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.2.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.2.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/krb5kdc/oryx_127.2.2.1.keytab.
Entry for principal oryx/127.2.2.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/krb5kdc/oryx_127.2.2.1.keytab.
I20260619 14:32:36.708810 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.1:0
--local_ip_for_outbound_sockets=127.2.2.1
--webserver_interface=127.2.2.1
--webserver_port=0
--tserver_master_addrs=127.2.2.62:38669
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/krb5kdc/oryx_127.2.2.1.keytab
--principal=oryx/127.2.2.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:36765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:32:36.835990 9669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:36.836279 9669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:36.836326 9669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:36.840441 9669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:36.840516 9669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:36.840559 9669 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 14:32:36.840633 9669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.1
I20260619 14:32:36.847019 9669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:36765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/krb5kdc/oryx_127.2.2.1.keytab
--principal=oryx/127.2.2.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.2.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.2.62:38669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.1
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:32:36.848210 9669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:36.849300 9669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:32:36.859076 9677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:36.859162 9675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:36.859169 9674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:36.859234 9669 server_base.cc:1034] running on GCE node
I20260619 14:32:36.859763 9669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:36.860397 9669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:36.861572 9669 hybrid_clock.cc:648] HybridClock initialized: now 1781879556861551 us; error 29 us; skew 500 ppm
Jun 19 14:32:36 dist-test-slave-70rm krb5kdc[9552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879556, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.2.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:32:36.864691 9669 init.cc:377] Logged in from keytab as oryx/127.2.2.1@KRBTEST.COM (short username oryx)
I20260619 14:32:36.865720 9669 webserver.cc:460] Webserver started at http://127.2.2.1:45499/ using document root <none> and password file <none>
I20260619 14:32:36.866381 9669 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:36.866483 9669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:36.866734 9669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:32:36.868911 9669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/ts-0/data/instance:
uuid: "db3354f894ef40e3a566e882876af4f4"
format_stamp: "Formatted at 2026-06-19 14:32:36 on dist-test-slave-70rm"
server_key: "7c6509bb8f00b2a59735c1da7f996a80"
server_key_iv: "4048e9ce5daf8170623e5ca4403aa200"
server_key_version: "encryptionkey@0"
I20260619 14:32:36.869539 9669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/ts-0/wal/instance:
uuid: "db3354f894ef40e3a566e882876af4f4"
format_stamp: "Formatted at 2026-06-19 14:32:36 on dist-test-slave-70rm"
server_key: "7c6509bb8f00b2a59735c1da7f996a80"
server_key_iv: "4048e9ce5daf8170623e5ca4403aa200"
server_key_version: "encryptionkey@0"
I20260619 14:32:36.873682 9669 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 14:32:36.876437 9684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:36.877797 9669 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:32:36.877945 9669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/ts-0/wal
uuid: "db3354f894ef40e3a566e882876af4f4"
format_stamp: "Formatted at 2026-06-19 14:32:36 on dist-test-slave-70rm"
server_key: "7c6509bb8f00b2a59735c1da7f996a80"
server_key_iv: "4048e9ce5daf8170623e5ca4403aa200"
server_key_version: "encryptionkey@0"
I20260619 14:32:36.878072 9669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:36.904047 9669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:36.907773 9669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:32:36.907996 9669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:36.909494 9669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:32:36.909572 9669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:36.909636 9669 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:32:36.909687 9669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 14:32:36 dist-test-slave-70rm krb5kdc[9552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879556, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.2.1@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM
I20260619 14:32:36.920845 9669 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.1:37821
I20260619 14:32:36.921010 9799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.1:37821 every 8 connection(s)
I20260619 14:32:36.921968 9669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
I20260619 14:32:36.925282 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:36.910450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:35649 (local address 127.2.2.62:38669)
0619 14:32:36.910576 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:36.910583 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:36.911703 (+ 1120us) server_negotiation.cc:407] Connection header received
0619 14:32:36.912898 (+ 1195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:36.912902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:36.912962 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:36.913078 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:36.914875 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:36.915761 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:36.917798 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:36.918033 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:36.918170 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:36.921451 (+ 3281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:36.921454 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:36.921457 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:36.921484 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:36.923154 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:36.923740 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:36.923744 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:36.923746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:36.923834 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:36.924179 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:36.924183 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:36.924185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:36.924380 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:36.924483 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:36.924984 (+ 501us) server_negotiation.cc:299] Negotiation successful
0619 14:32:36.925108 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 14:32:36.926138 9693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:36.911068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:38669 (local address 127.2.2.1:35649)
0619 14:32:36.911552 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:36.911594 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:36.912649 (+ 1055us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:36.913322 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:36.913333 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:36.913846 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:36.914674 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:36.914700 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:36.915902 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:36.915908 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:36.917650 (+ 1742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:36.917661 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:36.918168 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:36.918238 (+ 70us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:36.918468 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:36.919273 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:36.919302 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:36.921263 (+ 1961us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:36.923277 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:36.923283 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:36.923298 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:36.923624 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:36.923929 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:36.923932 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:36.923934 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:36.924078 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:36.924493 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:36.924501 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:36.924748 (+ 247us) client_negotiation.cc:769] Sending connection context
0619 14:32:36.924932 (+ 184us) client_negotiation.cc:240] Negotiation successful
0619 14:32:36.925265 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":138,"threads_started":1}
I20260619 14:32:36.926366 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 9669
I20260619 14:32:36.926481 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/ts-0/wal/instance
I20260619 14:32:36.926796 2056 external_mini_cluster.cc:1428] Setting key 564f2391a52a988fbd1febf055b340aa
I20260619 14:32:36.927440 9800 heartbeater.cc:347] Connected to a master server at 127.2.2.62:38669
I20260619 14:32:36.927609 9800 heartbeater.cc:464] Registering TS with master...
I20260619 14:32:36.928153 9800 heartbeater.cc:511] Master 127.2.2.62:38669 requested a full tablet report, sending...
I20260619 14:32:36.929828 9591 ts_manager.cc:194] Registered new tserver with Master: db3354f894ef40e3a566e882876af4f4 (127.2.2.1:37821)
I20260619 14:32:36.931216 9591 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.2.1@KRBTEST.COM'} at 127.2.2.1:35649
I20260619 14:32:36.937211 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:36.929663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:46637 (local address 127.2.2.62:38669)
0619 14:32:36.929788 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:36.929792 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:36.930380 (+ 588us) server_negotiation.cc:407] Connection header received
0619 14:32:36.930461 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:36.930464 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:36.930506 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:36.930584 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:36.931162 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:36.931845 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:36.932702 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:36.932938 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:36.933048 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:36.934208 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:36.934212 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:36.934215 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:36.934239 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:36.935642 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:36.936049 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:36.936054 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:36.936056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:36.936136 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:36.936407 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:36.936412 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:36.936413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:36.936585 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:36.936688 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:36.936942 (+ 254us) server_negotiation.cc:299] Negotiation successful
0619 14:32:36.937050 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 14:32:36.937218 9806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:36.929617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:38669 (local address 127.2.2.1:46637)
0619 14:32:36.930289 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:36.930301 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:36.930383 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:36.930625 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:36.930631 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:36.930883 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:36.931034 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:36.931043 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:36.931951 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:36.931955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:36.932584 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:36.932593 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:36.933036 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:36.933053 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:36.933142 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:36.933708 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:36.933722 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:36.934055 (+ 333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:36.935750 (+ 1695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:36.935753 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:36.935756 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:36.935953 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:36.936224 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:36.936229 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:36.936231 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:36.936321 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:36.936677 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:36.936680 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:36.936776 (+ 96us) client_negotiation.cc:769] Sending connection context
0619 14:32:36.936879 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 14:32:36.937022 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":93,"threads_started":1}
WARNING: no policy specified for oryx/127.2.2.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.2.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.2.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/krb5kdc/oryx_127.2.2.2.keytab.
Entry for principal oryx/127.2.2.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/krb5kdc/oryx_127.2.2.2.keytab.
I20260619 14:32:36.957463 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.2:0
--local_ip_for_outbound_sockets=127.2.2.2
--webserver_interface=127.2.2.2
--webserver_port=0
--tserver_master_addrs=127.2.2.62:38669
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/krb5kdc/oryx_127.2.2.2.keytab
--principal=oryx/127.2.2.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:36765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:32:37.073477 9809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:37.073810 9809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:37.073905 9809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:37.078140 9809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:37.078254 9809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:37.078321 9809 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 14:32:37.078404 9809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.2
I20260619 14:32:37.083979 9809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:36765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/krb5kdc/oryx_127.2.2.2.keytab
--principal=oryx/127.2.2.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.2.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.2.62:38669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.2
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:32:37.085047 9809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:37.086042 9809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:32:37.096323 9814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:37.096312 9817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:37.096376 9809 server_base.cc:1034] running on GCE node
W20260619 14:32:37.096318 9815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:37.096997 9809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:37.097714 9809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:37.098906 9809 hybrid_clock.cc:648] HybridClock initialized: now 1781879557098889 us; error 51 us; skew 500 ppm
Jun 19 14:32:37 dist-test-slave-70rm krb5kdc[9552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879557, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.2.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:32:37.101917 9809 init.cc:377] Logged in from keytab as oryx/127.2.2.2@KRBTEST.COM (short username oryx)
I20260619 14:32:37.102953 9809 webserver.cc:460] Webserver started at http://127.2.2.2:42089/ using document root <none> and password file <none>
I20260619 14:32:37.103543 9809 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:37.103637 9809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:37.103879 9809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:32:37.105993 9809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/ts-1/data/instance:
uuid: "625146abfd2c4086badda901ef08c98c"
format_stamp: "Formatted at 2026-06-19 14:32:37 on dist-test-slave-70rm"
server_key: "b984fb1a17ce98b69be2d1d2f37fdb16"
server_key_iv: "9276408b9e8774cd567ec5af4f49055e"
server_key_version: "encryptionkey@0"
I20260619 14:32:37.106662 9809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/ts-1/wal/instance:
uuid: "625146abfd2c4086badda901ef08c98c"
format_stamp: "Formatted at 2026-06-19 14:32:37 on dist-test-slave-70rm"
server_key: "b984fb1a17ce98b69be2d1d2f37fdb16"
server_key_iv: "9276408b9e8774cd567ec5af4f49055e"
server_key_version: "encryptionkey@0"
I20260619 14:32:37.110711 9809 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 14:32:37.113545 9824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:37.115046 9809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:32:37.115222 9809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/ts-1/wal
uuid: "625146abfd2c4086badda901ef08c98c"
format_stamp: "Formatted at 2026-06-19 14:32:37 on dist-test-slave-70rm"
server_key: "b984fb1a17ce98b69be2d1d2f37fdb16"
server_key_iv: "9276408b9e8774cd567ec5af4f49055e"
server_key_version: "encryptionkey@0"
I20260619 14:32:37.115368 9809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:37.126917 9809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:37.130934 9809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:32:37.131176 9809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:37.132838 9809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:32:37.132931 9809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:37.133050 9809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:32:37.133090 9809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 14:32:37 dist-test-slave-70rm krb5kdc[9552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879557, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.2.2@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM
I20260619 14:32:37.145856 9809 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.2:40949
I20260619 14:32:37.147092 9809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
I20260619 14:32:37.148293 9939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.2:40949 every 8 connection(s)
I20260619 14:32:37.149073 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:37.133642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:47175 (local address 127.2.2.62:38669)
0619 14:32:37.133822 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:37.133827 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:37.134934 (+ 1107us) server_negotiation.cc:407] Connection header received
0619 14:32:37.136400 (+ 1466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:37.136409 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:37.136487 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:37.136558 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:37.138877 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:37.139502 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:37.140703 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:37.140972 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:37.141078 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:37.144574 (+ 3496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:37.144579 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:37.144582 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:37.144616 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:37.146500 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:37.147062 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:37.147066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:37.147068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:37.147179 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:37.147831 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:37.147836 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:37.147839 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:37.148111 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:37.148211 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:37.148780 (+ 569us) server_negotiation.cc:299] Negotiation successful
0619 14:32:37.148907 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 14:32:37.149883 9833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:37.134098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:38669 (local address 127.2.2.2:47175)
0619 14:32:37.134714 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:37.134784 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:37.136084 (+ 1300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:37.137092 (+ 1008us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:37.137107 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:37.137703 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:37.138644 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:37.138675 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:37.139641 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:37.139644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:37.140565 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:37.140575 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:37.141113 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:37.141188 (+ 75us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:37.141464 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:37.142294 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:37.142316 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:37.144376 (+ 2060us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:37.146630 (+ 2254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:37.146638 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:37.146655 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:37.146944 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:37.147299 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:37.147302 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:37.147304 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:37.147499 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:37.148209 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:37.148216 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:37.148531 (+ 315us) client_negotiation.cc:769] Sending connection context
0619 14:32:37.148717 (+ 186us) client_negotiation.cc:240] Negotiation successful
0619 14:32:37.149049 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":113,"threads_started":1}
I20260619 14:32:37.151343 9940 heartbeater.cc:347] Connected to a master server at 127.2.2.62:38669
I20260619 14:32:37.151520 9940 heartbeater.cc:464] Registering TS with master...
I20260619 14:32:37.152151 9940 heartbeater.cc:511] Master 127.2.2.62:38669 requested a full tablet report, sending...
I20260619 14:32:37.153215 9597 ts_manager.cc:194] Registered new tserver with Master: 625146abfd2c4086badda901ef08c98c (127.2.2.2:40949)
I20260619 14:32:37.154031 9597 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.2.2@KRBTEST.COM'} at 127.2.2.2:47175
I20260619 14:32:37.154301 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 9809
I20260619 14:32:37.154417 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/ts-1/wal/instance
I20260619 14:32:37.154728 2056 external_mini_cluster.cc:1428] Setting key 93aed1303de4b29cb1c8fbf8d955f13c
I20260619 14:32:37.161892 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:37.153495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:43273 (local address 127.2.2.62:38669)
0619 14:32:37.153627 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:37.153631 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:37.153808 (+ 177us) server_negotiation.cc:407] Connection header received
0619 14:32:37.153917 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:37.153920 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:37.153957 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:37.154062 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:37.154591 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:37.155317 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:37.156437 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:37.156841 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:37.156963 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:37.158481 (+ 1518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:37.158484 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:37.158486 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:37.158509 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:37.160046 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:37.160597 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:37.160599 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:37.160600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:37.160675 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:37.161008 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:37.161011 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:37.161012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:37.161217 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:37.161315 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:37.161662 (+ 347us) server_negotiation.cc:299] Negotiation successful
0619 14:32:37.161771 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 14:32:37.162174 9946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:37.153428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:38669 (local address 127.2.2.2:43273)
0619 14:32:37.153718 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:37.153733 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:37.153836 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:37.154095 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:37.154101 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:37.154350 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:37.154470 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:37.154476 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:37.155474 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:37.155478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:37.156308 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:37.156318 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:37.156963 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:37.156979 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:37.157088 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:37.157661 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:37.157685 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:37.158319 (+ 634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:37.160176 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:37.160183 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:37.160187 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:37.160488 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:37.160771 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:37.160777 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:37.160782 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:37.160905 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:37.161336 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:37.161342 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:37.161478 (+ 136us) client_negotiation.cc:769] Sending connection context
0619 14:32:37.161623 (+ 145us) client_negotiation.cc:240] Negotiation successful
0619 14:32:37.161823 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":3,"thread_start_us":120,"threads_started":1}
WARNING: no policy specified for oryx/127.2.2.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.2.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.2.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/krb5kdc/oryx_127.2.2.3.keytab.
Entry for principal oryx/127.2.2.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/krb5kdc/oryx_127.2.2.3.keytab.
I20260619 14:32:37.183614 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.3:0
--local_ip_for_outbound_sockets=127.2.2.3
--webserver_interface=127.2.2.3
--webserver_port=0
--tserver_master_addrs=127.2.2.62:38669
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/krb5kdc/oryx_127.2.2.3.keytab
--principal=oryx/127.2.2.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:36765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:32:37.294845 9949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:37.295162 9949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:37.295241 9949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:37.299353 9949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:37.299453 9949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:37.299530 9949 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 14:32:37.299616 9949 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.3
I20260619 14:32:37.304924 9949 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:36765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/krb5kdc/oryx_127.2.2.3.keytab
--principal=oryx/127.2.2.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.2.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.2.62:38669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.3
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:32:37.306053 9949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:37.307169 9949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:32:37.317065 9954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:37.317065 9955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:37.317133 9957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:37.317231 9949 server_base.cc:1034] running on GCE node
I20260619 14:32:37.317816 9949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:37.318617 9949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:37.319815 9949 hybrid_clock.cc:648] HybridClock initialized: now 1781879557319742 us; error 89 us; skew 500 ppm
Jun 19 14:32:37 dist-test-slave-70rm krb5kdc[9552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879557, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.2.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:32:37.322857 9949 init.cc:377] Logged in from keytab as oryx/127.2.2.3@KRBTEST.COM (short username oryx)
I20260619 14:32:37.323724 9949 webserver.cc:460] Webserver started at http://127.2.2.3:34225/ using document root <none> and password file <none>
I20260619 14:32:37.324304 9949 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:37.324395 9949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:37.324621 9949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:32:37.326756 9949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/ts-2/data/instance:
uuid: "d59bfc2b333a46bf94f4b782a7e388cd"
format_stamp: "Formatted at 2026-06-19 14:32:37 on dist-test-slave-70rm"
server_key: "a11995317e9dcf3592c9cd1fd523a0ab"
server_key_iv: "194b9e611aba8de09135037638bbdd64"
server_key_version: "encryptionkey@0"
I20260619 14:32:37.327381 9949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/ts-2/wal/instance:
uuid: "d59bfc2b333a46bf94f4b782a7e388cd"
format_stamp: "Formatted at 2026-06-19 14:32:37 on dist-test-slave-70rm"
server_key: "a11995317e9dcf3592c9cd1fd523a0ab"
server_key_iv: "194b9e611aba8de09135037638bbdd64"
server_key_version: "encryptionkey@0"
I20260619 14:32:37.331213 9949 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:32:37.333847 9964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:37.335175 9949 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:32:37.335333 9949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/ts-2/wal
uuid: "d59bfc2b333a46bf94f4b782a7e388cd"
format_stamp: "Formatted at 2026-06-19 14:32:37 on dist-test-slave-70rm"
server_key: "a11995317e9dcf3592c9cd1fd523a0ab"
server_key_iv: "194b9e611aba8de09135037638bbdd64"
server_key_version: "encryptionkey@0"
I20260619 14:32:37.335451 9949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:37.359664 9949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:37.362831 9949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:32:37.363046 9949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:37.364501 9949 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:32:37.364578 9949 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:37.364655 9949 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:32:37.364704 9949 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 14:32:37 dist-test-slave-70rm krb5kdc[9552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879557, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.2.3@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM
I20260619 14:32:37.375602 9949 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.3:44701
I20260619 14:32:37.375629 10079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.3:44701 every 8 connection(s)
I20260619 14:32:37.376681 9949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
I20260619 14:32:37.378789 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:37.365431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.3:54485 (local address 127.2.2.62:38669)
0619 14:32:37.365597 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:37.365602 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:37.366513 (+ 911us) server_negotiation.cc:407] Connection header received
0619 14:32:37.367575 (+ 1062us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:37.367579 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:37.367637 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:37.367696 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:37.369584 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:37.370383 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:37.371570 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:37.371991 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:37.372129 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:37.374881 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:37.374884 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:37.374886 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:37.374912 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:37.376537 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:37.377178 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:37.377182 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:37.377185 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:37.377302 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:37.377655 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:37.377659 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:37.377661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:37.377915 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:37.378002 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:37.378537 (+ 535us) server_negotiation.cc:299] Negotiation successful
0619 14:32:37.378657 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 14:32:37.379592 9973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:37.365848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:38669 (local address 127.2.2.3:54485)
0619 14:32:37.366370 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:37.366420 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:37.367298 (+ 878us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:37.368192 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:37.368202 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:37.368668 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:37.369406 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:37.369430 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:37.370511 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:37.370515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:37.371410 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:37.371419 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:37.372126 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:37.372190 (+ 64us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:37.372421 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:37.373076 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:37.373093 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:37.374696 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:37.376673 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:37.376679 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:37.376695 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:37.377042 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:37.377405 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:37.377408 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:37.377412 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:37.377543 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:37.378009 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:37.378014 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:37.378322 (+ 308us) client_negotiation.cc:769] Sending connection context
0619 14:32:37.378476 (+ 154us) client_negotiation.cc:240] Negotiation successful
0619 14:32:37.378729 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":137,"threads_started":1}
I20260619 14:32:37.380237 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 9949
I20260619 14:32:37.380379 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/ts-2/wal/instance
I20260619 14:32:37.380677 2056 external_mini_cluster.cc:1428] Setting key 8b33bf1b54b7e51fb8e3e735ff098a81
I20260619 14:32:37.380965 10080 heartbeater.cc:347] Connected to a master server at 127.2.2.62:38669
I20260619 14:32:37.381114 10080 heartbeater.cc:464] Registering TS with master...
I20260619 14:32:37.381799 10080 heartbeater.cc:511] Master 127.2.2.62:38669 requested a full tablet report, sending...
I20260619 14:32:37.382974 9597 ts_manager.cc:194] Registered new tserver with Master: d59bfc2b333a46bf94f4b782a7e388cd (127.2.2.3:44701)
I20260619 14:32:37.383723 9597 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.2.3@KRBTEST.COM'} at 127.2.2.3:54485
I20260619 14:32:37.390275 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:37.383483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.3:54731 (local address 127.2.2.62:38669)
0619 14:32:37.383608 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:37.383612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:37.383795 (+ 183us) server_negotiation.cc:407] Connection header received
0619 14:32:37.383887 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:37.383890 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:37.383934 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:37.384013 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:37.384488 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:37.384997 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:37.385805 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:37.386036 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:37.386164 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:37.387251 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:37.387255 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:37.387258 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:37.387282 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:37.388532 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:37.388984 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:37.388988 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:37.388990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:37.389080 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:37.389346 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:37.389350 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:37.389352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:37.389527 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:37.389633 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:37.389932 (+ 299us) server_negotiation.cc:299] Negotiation successful
0619 14:32:37.390059 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 14:32:37.390395 10086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:37.383373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:38669 (local address 127.2.2.3:54731)
0619 14:32:37.383716 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:37.383728 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:37.383808 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:37.384077 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:37.384080 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:37.384282 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:37.384377 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:37.384382 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:37.385119 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:37.385122 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:37.385685 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:37.385693 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:37.386164 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:37.386182 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:37.386280 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:37.386727 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:37.386739 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:37.387067 (+ 328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:37.388676 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:37.388679 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:37.388683 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:37.388884 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:37.389177 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:37.389180 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:37.389182 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:37.389261 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:37.389623 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:37.389626 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:37.389768 (+ 142us) client_negotiation.cc:769] Sending connection context
0619 14:32:37.389896 (+ 128us) client_negotiation.cc:240] Negotiation successful
0619 14:32:37.390057 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":113,"threads_started":1}
I20260619 14:32:37.394691 2056 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 14:32:37 dist-test-slave-70rm krb5kdc[9552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879557, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 14:32:37 dist-test-slave-70rm krb5kdc[9552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:32:37 dist-test-slave-70rm krb5kdc[9552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 14:32:37.418063 9646 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:32:37.411936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33080 (local address 127.2.2.62:38669)
0619 14:32:37.412075 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:37.412079 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:37.412091 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:32:37.412131 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:37.412133 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:37.412170 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:37.412263 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:37.413000 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:37.413524 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:37.414452 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:37.414676 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:37.414795 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:37.417946 (+ 3151us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33080: BlockingRecv error: recv got EOF from 127.0.0.1:33080 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 19 14:32:37 dist-test-slave-70rm krb5kdc[9552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879557, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM
I20260619 14:32:37.428087 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:37.419719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33088 (local address 127.2.2.62:38669)
0619 14:32:37.419843 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:37.419846 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:37.419992 (+ 146us) server_negotiation.cc:407] Connection header received
0619 14:32:37.420087 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:37.420089 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:37.420126 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:37.420192 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:37.420864 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:37.421361 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:37.422104 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:37.422377 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:37.422454 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:37.424891 (+ 2437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:37.424894 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:37.424896 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:37.424918 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:37.426306 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:37.426811 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:37.426814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:37.426815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:37.426906 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:37.427211 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:37.427213 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:37.427214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:37.427465 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:37.427569 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:37.427846 (+ 277us) server_negotiation.cc:299] Negotiation successful
0619 14:32:37.427940 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 14:32:37.431084 9597 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33088:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:32:37.434243 9597 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:32:37.448911 10104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:37.442786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.3:44701 (local address 127.0.0.1:50450)
0619 14:32:37.443927 (+ 1141us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:37.443942 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:37.444023 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:37.444727 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:37.444730 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:37.444748 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:32:37.444836 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:37.444842 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:37.446207 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:37.446211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:37.447544 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:37.447550 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:37.448349 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:37.448353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:37.448459 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:37.448494 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 14:32:37.448618 (+ 124us) client_negotiation.cc:240] Negotiation successful
0619 14:32:37.448749 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1045,"mutex_wait_us":60,"thread_start_us":96,"threads_started":1}
I20260619 14:32:37.448925 10106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:37.443383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50450 (local address 127.2.2.3:44701)
0619 14:32:37.444273 (+ 890us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:37.444285 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:37.444307 (+ 22us) server_negotiation.cc:407] Connection header received
0619 14:32:37.444382 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:37.444386 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:37.444585 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:37.444754 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:32:37.444945 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:37.446002 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:37.447652 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:37.448226 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:37.448352 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:37.448531 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:37.448648 (+ 117us) server_negotiation.cc:299] Negotiation successful
0619 14:32:37.448751 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":791,"thread_start_us":111,"threads_started":1}
I20260619 14:32:37.449283 10102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:37.442575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.2:40949 (local address 127.0.0.1:50778)
0619 14:32:37.443269 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:37.443320 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:37.443471 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:37.443999 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:37.444005 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:37.444028 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:32:37.444169 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:37.444175 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:37.445488 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:37.445495 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:37.447524 (+ 2029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:37.447535 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:37.448803 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:37.448807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:37.448962 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:37.448987 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 14:32:37.449062 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 14:32:37.449157 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"mutex_wait_us":56,"thread_start_us":93,"threads_started":1}
I20260619 14:32:37.449716 10103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:37.442743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50778 (local address 127.2.2.2:40949)
0619 14:32:37.443060 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:37.443072 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:37.443536 (+ 464us) server_negotiation.cc:407] Connection header received
0619 14:32:37.443622 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:37.443639 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:37.443840 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:37.444000 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:32:37.444281 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:37.445387 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:37.447673 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:37.448492 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:37.448597 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:37.448756 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:37.449467 (+ 711us) server_negotiation.cc:299] Negotiation successful
0619 14:32:37.449566 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":118,"threads_started":1}
I20260619 14:32:37.450213 10107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:37.443450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42494 (local address 127.2.2.1:37821)
0619 14:32:37.443767 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:37.443775 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:37.444502 (+ 727us) server_negotiation.cc:407] Connection header received
0619 14:32:37.444613 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:37.444618 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:37.444791 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:37.444919 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:32:37.445295 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:37.446419 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:37.447703 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:37.448492 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:37.448597 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:37.448737 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:37.449961 (+ 1224us) server_negotiation.cc:299] Negotiation successful
0619 14:32:37.450062 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":73,"threads_started":1}
I20260619 14:32:37.450680 10105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:37.443296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.1:37821 (local address 127.0.0.1:42494)
0619 14:32:37.444414 (+ 1118us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:37.444426 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:37.444509 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:37.445045 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:37.445048 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:37.445064 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:32:37.445147 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:37.445152 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:37.446559 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:37.446563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:37.447581 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:37.447611 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:37.448803 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:37.448807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:37.448966 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:37.449327 (+ 361us) client_negotiation.cc:769] Sending connection context
0619 14:32:37.449618 (+ 291us) client_negotiation.cc:240] Negotiation successful
0619 14:32:37.449700 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1065,"mutex_wait_us":50,"thread_start_us":101,"threads_started":1}
I20260619 14:32:37.452247 10014 tablet_service.cc:1467] Processing CreateTablet for tablet 6fa96f1330b942d68fd14dad9c69a527 (DEFAULT_TABLE table=test-table [id=00113256b29d42e2ba0a22c090e3c693]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:32:37.452405 9874 tablet_service.cc:1467] Processing CreateTablet for tablet 6fa96f1330b942d68fd14dad9c69a527 (DEFAULT_TABLE table=test-table [id=00113256b29d42e2ba0a22c090e3c693]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:32:37.453019 9734 tablet_service.cc:1467] Processing CreateTablet for tablet 6fa96f1330b942d68fd14dad9c69a527 (DEFAULT_TABLE table=test-table [id=00113256b29d42e2ba0a22c090e3c693]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:32:37.453369 9874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fa96f1330b942d68fd14dad9c69a527. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:37.453379 10014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fa96f1330b942d68fd14dad9c69a527. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:37.453886 9734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fa96f1330b942d68fd14dad9c69a527. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:37.460614 10108 tablet_bootstrap.cc:492] T 6fa96f1330b942d68fd14dad9c69a527 P db3354f894ef40e3a566e882876af4f4: Bootstrap starting.
I20260619 14:32:37.461719 10110 tablet_bootstrap.cc:492] T 6fa96f1330b942d68fd14dad9c69a527 P d59bfc2b333a46bf94f4b782a7e388cd: Bootstrap starting.
I20260619 14:32:37.462066 10109 tablet_bootstrap.cc:492] T 6fa96f1330b942d68fd14dad9c69a527 P 625146abfd2c4086badda901ef08c98c: Bootstrap starting.
I20260619 14:32:37.462818 10108 tablet_bootstrap.cc:654] T 6fa96f1330b942d68fd14dad9c69a527 P db3354f894ef40e3a566e882876af4f4: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:37.463555 10108 log.cc:824] T 6fa96f1330b942d68fd14dad9c69a527 P db3354f894ef40e3a566e882876af4f4: Log is configured to *not* fsync() on all Append() calls
I20260619 14:32:37.463878 10110 tablet_bootstrap.cc:654] T 6fa96f1330b942d68fd14dad9c69a527 P d59bfc2b333a46bf94f4b782a7e388cd: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:37.463948 10109 tablet_bootstrap.cc:654] T 6fa96f1330b942d68fd14dad9c69a527 P 625146abfd2c4086badda901ef08c98c: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:37.464751 10110 log.cc:824] T 6fa96f1330b942d68fd14dad9c69a527 P d59bfc2b333a46bf94f4b782a7e388cd: Log is configured to *not* fsync() on all Append() calls
I20260619 14:32:37.464754 10109 log.cc:824] T 6fa96f1330b942d68fd14dad9c69a527 P 625146abfd2c4086badda901ef08c98c: Log is configured to *not* fsync() on all Append() calls
I20260619 14:32:37.465680 10108 tablet_bootstrap.cc:492] T 6fa96f1330b942d68fd14dad9c69a527 P db3354f894ef40e3a566e882876af4f4: No bootstrap required, opened a new log
I20260619 14:32:37.465878 10108 ts_tablet_manager.cc:1397] T 6fa96f1330b942d68fd14dad9c69a527 P db3354f894ef40e3a566e882876af4f4: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260619 14:32:37.466769 10110 tablet_bootstrap.cc:492] T 6fa96f1330b942d68fd14dad9c69a527 P d59bfc2b333a46bf94f4b782a7e388cd: No bootstrap required, opened a new log
I20260619 14:32:37.466959 10110 ts_tablet_manager.cc:1397] T 6fa96f1330b942d68fd14dad9c69a527 P d59bfc2b333a46bf94f4b782a7e388cd: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:32:37.467525 10109 tablet_bootstrap.cc:492] T 6fa96f1330b942d68fd14dad9c69a527 P 625146abfd2c4086badda901ef08c98c: No bootstrap required, opened a new log
I20260619 14:32:37.467725 10109 ts_tablet_manager.cc:1397] T 6fa96f1330b942d68fd14dad9c69a527 P 625146abfd2c4086badda901ef08c98c: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260619 14:32:37.470605 10109 raft_consensus.cc:348] T 6fa96f1330b942d68fd14dad9c69a527 P 625146abfd2c4086badda901ef08c98c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625146abfd2c4086badda901ef08c98c" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 40949 } } peers { permanent_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44701 } } peers { permanent_uuid: "db3354f894ef40e3a566e882876af4f4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37821 } }
I20260619 14:32:37.470654 10108 raft_consensus.cc:348] T 6fa96f1330b942d68fd14dad9c69a527 P db3354f894ef40e3a566e882876af4f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625146abfd2c4086badda901ef08c98c" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 40949 } } peers { permanent_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44701 } } peers { permanent_uuid: "db3354f894ef40e3a566e882876af4f4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37821 } }
I20260619 14:32:37.470893 10109 raft_consensus.cc:374] T 6fa96f1330b942d68fd14dad9c69a527 P 625146abfd2c4086badda901ef08c98c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:37.470942 10109 raft_consensus.cc:729] T 6fa96f1330b942d68fd14dad9c69a527 P 625146abfd2c4086badda901ef08c98c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 625146abfd2c4086badda901ef08c98c, State: Initialized, Role: FOLLOWER
I20260619 14:32:37.470942 10108 raft_consensus.cc:374] T 6fa96f1330b942d68fd14dad9c69a527 P db3354f894ef40e3a566e882876af4f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:37.470997 10108 raft_consensus.cc:729] T 6fa96f1330b942d68fd14dad9c69a527 P db3354f894ef40e3a566e882876af4f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db3354f894ef40e3a566e882876af4f4, State: Initialized, Role: FOLLOWER
I20260619 14:32:37.470949 10110 raft_consensus.cc:348] T 6fa96f1330b942d68fd14dad9c69a527 P d59bfc2b333a46bf94f4b782a7e388cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625146abfd2c4086badda901ef08c98c" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 40949 } } peers { permanent_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44701 } } peers { permanent_uuid: "db3354f894ef40e3a566e882876af4f4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37821 } }
I20260619 14:32:37.471174 10110 raft_consensus.cc:374] T 6fa96f1330b942d68fd14dad9c69a527 P d59bfc2b333a46bf94f4b782a7e388cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:37.471240 10110 raft_consensus.cc:729] T 6fa96f1330b942d68fd14dad9c69a527 P d59bfc2b333a46bf94f4b782a7e388cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d59bfc2b333a46bf94f4b782a7e388cd, State: Initialized, Role: FOLLOWER
I20260619 14:32:37.471436 10109 consensus_queue.cc:260] T 6fa96f1330b942d68fd14dad9c69a527 P 625146abfd2c4086badda901ef08c98c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625146abfd2c4086badda901ef08c98c" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 40949 } } peers { permanent_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44701 } } peers { permanent_uuid: "db3354f894ef40e3a566e882876af4f4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37821 } }
I20260619 14:32:37.471480 10108 consensus_queue.cc:260] T 6fa96f1330b942d68fd14dad9c69a527 P db3354f894ef40e3a566e882876af4f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625146abfd2c4086badda901ef08c98c" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 40949 } } peers { permanent_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44701 } } peers { permanent_uuid: "db3354f894ef40e3a566e882876af4f4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37821 } }
I20260619 14:32:37.471580 10110 consensus_queue.cc:260] T 6fa96f1330b942d68fd14dad9c69a527 P d59bfc2b333a46bf94f4b782a7e388cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625146abfd2c4086badda901ef08c98c" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 40949 } } peers { permanent_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44701 } } peers { permanent_uuid: "db3354f894ef40e3a566e882876af4f4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37821 } }
I20260619 14:32:37.472499 10109 ts_tablet_manager.cc:1428] T 6fa96f1330b942d68fd14dad9c69a527 P 625146abfd2c4086badda901ef08c98c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 14:32:37.472502 10108 ts_tablet_manager.cc:1428] T 6fa96f1330b942d68fd14dad9c69a527 P db3354f894ef40e3a566e882876af4f4: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20260619 14:32:37.472501 10110 ts_tablet_manager.cc:1428] T 6fa96f1330b942d68fd14dad9c69a527 P d59bfc2b333a46bf94f4b782a7e388cd: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 14:32:37.472851 9800 heartbeater.cc:503] Master 127.2.2.62:38669 was elected leader, sending a full tablet report...
I20260619 14:32:37.472851 10080 heartbeater.cc:503] Master 127.2.2.62:38669 was elected leader, sending a full tablet report...
I20260619 14:32:37.472852 9940 heartbeater.cc:503] Master 127.2.2.62:38669 was elected leader, sending a full tablet report...
I20260619 14:32:37.542896 10115 raft_consensus.cc:482] T 6fa96f1330b942d68fd14dad9c69a527 P 625146abfd2c4086badda901ef08c98c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:32:37.543139 10115 raft_consensus.cc:504] T 6fa96f1330b942d68fd14dad9c69a527 P 625146abfd2c4086badda901ef08c98c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625146abfd2c4086badda901ef08c98c" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 40949 } } peers { permanent_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44701 } } peers { permanent_uuid: "db3354f894ef40e3a566e882876af4f4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37821 } }
I20260619 14:32:37.544315 10115 leader_election.cc:288] T 6fa96f1330b942d68fd14dad9c69a527 P 625146abfd2c4086badda901ef08c98c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d59bfc2b333a46bf94f4b782a7e388cd (127.2.2.3:44701), db3354f894ef40e3a566e882876af4f4 (127.2.2.1:37821)
I20260619 14:32:37.549304 10106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:37.544780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:36837 (local address 127.2.2.3:44701)
0619 14:32:37.544961 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:37.544966 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:37.544983 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:32:37.545137 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:37.545141 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:37.545189 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:37.545256 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:32:37.545783 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:37.546706 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:37.548171 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:37.548775 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:37.548877 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:37.548948 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:37.549133 (+ 185us) server_negotiation.cc:299] Negotiation successful
0619 14:32:37.549197 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 14:32:37.549327 9833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:37.544686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.3:44701 (local address 127.2.2.2:36837)
0619 14:32:37.544835 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:37.544853 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:37.544988 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:37.545387 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:37.545390 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:37.545497 (+ 107us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:32:37.545641 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:37.545648 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:37.546820 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:37.546823 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:37.547945 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:37.547954 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:37.548891 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:37.548894 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:37.548998 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:37.549023 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 14:32:37.549107 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 14:32:37.549181 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"spinlock_wait_cycles":121472}
I20260619 14:32:37.550035 10107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:37.545187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:42763 (local address 127.2.2.1:37821)
0619 14:32:37.545307 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:37.545312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:37.545427 (+ 115us) server_negotiation.cc:407] Connection header received
0619 14:32:37.545623 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:37.545627 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:37.545679 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:37.545759 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:32:37.546685 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:37.547486 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:37.548876 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:37.549499 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:37.549618 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:37.549685 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:37.549863 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 14:32:37.549941 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 14:32:37.550056 10117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:37.544980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.1:37821 (local address 127.2.2.2:42763)
0619 14:32:37.545337 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:37.545349 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:37.545509 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:37.545783 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:37.545785 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:37.545803 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:32:37.546519 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:37.546531 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:37.547591 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:37.547595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:37.548710 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:37.548722 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:37.549612 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:37.549616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:37.549723 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:37.549752 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 14:32:37.549817 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 14:32:37.549908 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"spinlock_wait_cycles":7296,"thread_start_us":178,"threads_started":1}
I20260619 14:32:37.550225 10034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa96f1330b942d68fd14dad9c69a527" candidate_uuid: "625146abfd2c4086badda901ef08c98c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" is_pre_election: true
I20260619 14:32:37.550547 10034 raft_consensus.cc:2454] T 6fa96f1330b942d68fd14dad9c69a527 P d59bfc2b333a46bf94f4b782a7e388cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 625146abfd2c4086badda901ef08c98c in term 0.
I20260619 14:32:37.551041 9754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa96f1330b942d68fd14dad9c69a527" candidate_uuid: "625146abfd2c4086badda901ef08c98c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db3354f894ef40e3a566e882876af4f4" is_pre_election: true
I20260619 14:32:37.551105 9828 leader_election.cc:302] T 6fa96f1330b942d68fd14dad9c69a527 P 625146abfd2c4086badda901ef08c98c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 625146abfd2c4086badda901ef08c98c, d59bfc2b333a46bf94f4b782a7e388cd; no voters:
I20260619 14:32:37.551339 9754 raft_consensus.cc:2454] T 6fa96f1330b942d68fd14dad9c69a527 P db3354f894ef40e3a566e882876af4f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 625146abfd2c4086badda901ef08c98c in term 0.
I20260619 14:32:37.551436 10115 raft_consensus.cc:2781] T 6fa96f1330b942d68fd14dad9c69a527 P 625146abfd2c4086badda901ef08c98c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:32:37.551530 10115 raft_consensus.cc:482] T 6fa96f1330b942d68fd14dad9c69a527 P 625146abfd2c4086badda901ef08c98c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:32:37.551698 10115 raft_consensus.cc:3037] T 6fa96f1330b942d68fd14dad9c69a527 P 625146abfd2c4086badda901ef08c98c [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:37.552755 10115 raft_consensus.cc:504] T 6fa96f1330b942d68fd14dad9c69a527 P 625146abfd2c4086badda901ef08c98c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625146abfd2c4086badda901ef08c98c" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 40949 } } peers { permanent_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44701 } } peers { permanent_uuid: "db3354f894ef40e3a566e882876af4f4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37821 } }
I20260619 14:32:37.553051 10115 leader_election.cc:288] T 6fa96f1330b942d68fd14dad9c69a527 P 625146abfd2c4086badda901ef08c98c [CANDIDATE]: Term 1 election: Requested vote from peers d59bfc2b333a46bf94f4b782a7e388cd (127.2.2.3:44701), db3354f894ef40e3a566e882876af4f4 (127.2.2.1:37821)
I20260619 14:32:37.553468 10034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa96f1330b942d68fd14dad9c69a527" candidate_uuid: "625146abfd2c4086badda901ef08c98c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d59bfc2b333a46bf94f4b782a7e388cd"
I20260619 14:32:37.553575 9754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa96f1330b942d68fd14dad9c69a527" candidate_uuid: "625146abfd2c4086badda901ef08c98c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db3354f894ef40e3a566e882876af4f4"
I20260619 14:32:37.553624 10034 raft_consensus.cc:3037] T 6fa96f1330b942d68fd14dad9c69a527 P d59bfc2b333a46bf94f4b782a7e388cd [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:37.553740 9754 raft_consensus.cc:3037] T 6fa96f1330b942d68fd14dad9c69a527 P db3354f894ef40e3a566e882876af4f4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:37.554813 10034 raft_consensus.cc:2454] T 6fa96f1330b942d68fd14dad9c69a527 P d59bfc2b333a46bf94f4b782a7e388cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 625146abfd2c4086badda901ef08c98c in term 1.
I20260619 14:32:37.554826 9754 raft_consensus.cc:2454] T 6fa96f1330b942d68fd14dad9c69a527 P db3354f894ef40e3a566e882876af4f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 625146abfd2c4086badda901ef08c98c in term 1.
I20260619 14:32:37.555105 9828 leader_election.cc:302] T 6fa96f1330b942d68fd14dad9c69a527 P 625146abfd2c4086badda901ef08c98c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 625146abfd2c4086badda901ef08c98c, db3354f894ef40e3a566e882876af4f4; no voters:
I20260619 14:32:37.555285 10115 raft_consensus.cc:2781] T 6fa96f1330b942d68fd14dad9c69a527 P 625146abfd2c4086badda901ef08c98c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:32:37.555533 10115 raft_consensus.cc:686] T 6fa96f1330b942d68fd14dad9c69a527 P 625146abfd2c4086badda901ef08c98c [term 1 LEADER]: Becoming Leader. State: Replica: 625146abfd2c4086badda901ef08c98c, State: Running, Role: LEADER
I20260619 14:32:37.555892 10115 consensus_queue.cc:237] T 6fa96f1330b942d68fd14dad9c69a527 P 625146abfd2c4086badda901ef08c98c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625146abfd2c4086badda901ef08c98c" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 40949 } } peers { permanent_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44701 } } peers { permanent_uuid: "db3354f894ef40e3a566e882876af4f4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37821 } }
I20260619 14:32:37.559862 9597 catalog_manager.cc:5495] T 6fa96f1330b942d68fd14dad9c69a527 P 625146abfd2c4086badda901ef08c98c reported cstate change: term changed from 0 to 1, leader changed from <none> to 625146abfd2c4086badda901ef08c98c (127.2.2.2). New cstate: current_term: 1 leader_uuid: "625146abfd2c4086badda901ef08c98c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625146abfd2c4086badda901ef08c98c" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 40949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db3354f894ef40e3a566e882876af4f4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37821 } health_report { overall_health: UNKNOWN } } }
I20260619 14:32:37.571821 10103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:37.568424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50790 (local address 127.2.2.2:40949)
0619 14:32:37.568549 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:37.568552 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:37.568566 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:32:37.568678 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:37.568680 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:37.568732 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:37.568831 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:32:37.569036 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:37.569529 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:37.570465 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:37.570685 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:37.570769 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:37.571039 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:32:37.571462 (+ 423us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:32:37.571544 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:37.571639 (+ 95us) server_negotiation.cc:299] Negotiation successful
0619 14:32:37.571686 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 14:32:37.577409 9754 raft_consensus.cc:1261] T 6fa96f1330b942d68fd14dad9c69a527 P db3354f894ef40e3a566e882876af4f4 [term 1 FOLLOWER]: Refusing update from remote peer 625146abfd2c4086badda901ef08c98c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:32:37.577409 10034 raft_consensus.cc:1261] T 6fa96f1330b942d68fd14dad9c69a527 P d59bfc2b333a46bf94f4b782a7e388cd [term 1 FOLLOWER]: Refusing update from remote peer 625146abfd2c4086badda901ef08c98c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:32:37.578115 10115 consensus_queue.cc:1035] T 6fa96f1330b942d68fd14dad9c69a527 P 625146abfd2c4086badda901ef08c98c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:32:37.578402 10118 consensus_queue.cc:1035] T 6fa96f1330b942d68fd14dad9c69a527 P 625146abfd2c4086badda901ef08c98c [LEADER]: Connected to new peer: Peer: permanent_uuid: "db3354f894ef40e3a566e882876af4f4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:32:37.588320 10122 mvcc.cc:205] Tried to move back new op lower bound from 7298578667829882880 to 7298578667752189952. Current Snapshot: MvccSnapshot[applied={T|T < 7298578667829882880}]
I20260619 14:32:37.591012 10123 mvcc.cc:205] Tried to move back new op lower bound from 7298578667829882880 to 7298578667752189952. Current Snapshot: MvccSnapshot[applied={T|T < 7298578667829882880}]
I20260619 14:32:37.596292 9597 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:32:37.596477 9597 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:32:37.598901 9597 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 50728c18eca74389b5c824d5d291fa09: Sending DeleteTablet for 3 replicas of tablet 6fa96f1330b942d68fd14dad9c69a527
I20260619 14:32:37.599779 10014 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fa96f1330b942d68fd14dad9c69a527 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:32:37 UTC) from {username='oryx', principal='oryx/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:50450
I20260619 14:32:37.599956 9734 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fa96f1330b942d68fd14dad9c69a527 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:32:37 UTC) from {username='oryx', principal='oryx/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:42494
I20260619 14:32:37.600296 9874 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fa96f1330b942d68fd14dad9c69a527 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:32:37 UTC) from {username='oryx', principal='oryx/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:50778
I20260619 14:32:37.600373 9597 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33088:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 14:32:37.600787 10132 tablet_replica.cc:330] T 6fa96f1330b942d68fd14dad9c69a527 P d59bfc2b333a46bf94f4b782a7e388cd: stopping tablet replica
W20260619 14:32:37.600800 9597 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:32:37.601161 10132 raft_consensus.cc:2229] T 6fa96f1330b942d68fd14dad9c69a527 P d59bfc2b333a46bf94f4b782a7e388cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:32:37.601490 10132 raft_consensus.cc:2258] T 6fa96f1330b942d68fd14dad9c69a527 P d59bfc2b333a46bf94f4b782a7e388cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:32:37.601586 10133 tablet_replica.cc:330] T 6fa96f1330b942d68fd14dad9c69a527 P db3354f894ef40e3a566e882876af4f4: stopping tablet replica
I20260619 14:32:37.601866 10133 raft_consensus.cc:2229] T 6fa96f1330b942d68fd14dad9c69a527 P db3354f894ef40e3a566e882876af4f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:32:37.602170 10133 raft_consensus.cc:2258] T 6fa96f1330b942d68fd14dad9c69a527 P db3354f894ef40e3a566e882876af4f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:32:37.602391 10134 tablet_replica.cc:330] T 6fa96f1330b942d68fd14dad9c69a527 P 625146abfd2c4086badda901ef08c98c: stopping tablet replica
I20260619 14:32:37.602715 10134 raft_consensus.cc:2229] T 6fa96f1330b942d68fd14dad9c69a527 P 625146abfd2c4086badda901ef08c98c [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:32:37.603108 10134 raft_consensus.cc:2258] T 6fa96f1330b942d68fd14dad9c69a527 P 625146abfd2c4086badda901ef08c98c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:32:37.603699 10132 ts_tablet_manager.cc:1905] T 6fa96f1330b942d68fd14dad9c69a527 P d59bfc2b333a46bf94f4b782a7e388cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:32:37.604583 10134 ts_tablet_manager.cc:1905] T 6fa96f1330b942d68fd14dad9c69a527 P 625146abfd2c4086badda901ef08c98c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:32:37.606495 10132 ts_tablet_manager.cc:1918] T 6fa96f1330b942d68fd14dad9c69a527 P d59bfc2b333a46bf94f4b782a7e388cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:32:37.606596 10132 log.cc:1196] T 6fa96f1330b942d68fd14dad9c69a527 P d59bfc2b333a46bf94f4b782a7e388cd: Deleting WAL directory at /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/ts-2/wal/wals/6fa96f1330b942d68fd14dad9c69a527
I20260619 14:32:37.606913 10132 ts_tablet_manager.cc:1939] T 6fa96f1330b942d68fd14dad9c69a527 P d59bfc2b333a46bf94f4b782a7e388cd: Deleting consensus metadata
I20260619 14:32:37.607513 10134 ts_tablet_manager.cc:1918] T 6fa96f1330b942d68fd14dad9c69a527 P 625146abfd2c4086badda901ef08c98c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:32:37.607540 9734 tablet_service.cc:1467] Processing CreateTablet for tablet 722050258d2347f7b60552db8dae2fee (DEFAULT_TABLE table=test-table [id=ec5449b9956f4636a41f122ccbd18054]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:32:37.607724 10134 log.cc:1196] T 6fa96f1330b942d68fd14dad9c69a527 P 625146abfd2c4086badda901ef08c98c: Deleting WAL directory at /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/ts-1/wal/wals/6fa96f1330b942d68fd14dad9c69a527
I20260619 14:32:37.607836 9734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 722050258d2347f7b60552db8dae2fee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:37.608126 10134 ts_tablet_manager.cc:1939] T 6fa96f1330b942d68fd14dad9c69a527 P 625146abfd2c4086badda901ef08c98c: Deleting consensus metadata
I20260619 14:32:37.608964 10133 ts_tablet_manager.cc:1905] T 6fa96f1330b942d68fd14dad9c69a527 P db3354f894ef40e3a566e882876af4f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:32:37.610255 10108 tablet_bootstrap.cc:492] T 722050258d2347f7b60552db8dae2fee P db3354f894ef40e3a566e882876af4f4: Bootstrap starting.
I20260619 14:32:37.610486 10014 tablet_service.cc:1467] Processing CreateTablet for tablet 722050258d2347f7b60552db8dae2fee (DEFAULT_TABLE table=test-table [id=ec5449b9956f4636a41f122ccbd18054]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:32:37.610723 10014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 722050258d2347f7b60552db8dae2fee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:37.611308 10133 ts_tablet_manager.cc:1918] T 6fa96f1330b942d68fd14dad9c69a527 P db3354f894ef40e3a566e882876af4f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:32:37.611405 10133 log.cc:1196] T 6fa96f1330b942d68fd14dad9c69a527 P db3354f894ef40e3a566e882876af4f4: Deleting WAL directory at /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/ts-0/wal/wals/6fa96f1330b942d68fd14dad9c69a527
I20260619 14:32:37.611318 10108 tablet_bootstrap.cc:654] T 722050258d2347f7b60552db8dae2fee P db3354f894ef40e3a566e882876af4f4: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:37.611696 10133 ts_tablet_manager.cc:1939] T 6fa96f1330b942d68fd14dad9c69a527 P db3354f894ef40e3a566e882876af4f4: Deleting consensus metadata
I20260619 14:32:37.613098 10108 tablet_bootstrap.cc:492] T 722050258d2347f7b60552db8dae2fee P db3354f894ef40e3a566e882876af4f4: No bootstrap required, opened a new log
I20260619 14:32:37.613181 10108 ts_tablet_manager.cc:1397] T 722050258d2347f7b60552db8dae2fee P db3354f894ef40e3a566e882876af4f4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20260619 14:32:37.613219 10110 tablet_bootstrap.cc:492] T 722050258d2347f7b60552db8dae2fee P d59bfc2b333a46bf94f4b782a7e388cd: Bootstrap starting.
I20260619 14:32:37.613595 10108 raft_consensus.cc:348] T 722050258d2347f7b60552db8dae2fee P db3354f894ef40e3a566e882876af4f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db3354f894ef40e3a566e882876af4f4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37821 } } peers { permanent_uuid: "625146abfd2c4086badda901ef08c98c" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 40949 } } peers { permanent_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44701 } }
I20260619 14:32:37.613698 10108 raft_consensus.cc:374] T 722050258d2347f7b60552db8dae2fee P db3354f894ef40e3a566e882876af4f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:37.613652 9582 catalog_manager.cc:4841] TS 625146abfd2c4086badda901ef08c98c (127.2.2.2:40949): tablet 6fa96f1330b942d68fd14dad9c69a527 (table test-table [id=00113256b29d42e2ba0a22c090e3c693]) successfully deleted
I20260619 14:32:37.613727 10108 raft_consensus.cc:729] T 722050258d2347f7b60552db8dae2fee P db3354f894ef40e3a566e882876af4f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db3354f894ef40e3a566e882876af4f4, State: Initialized, Role: FOLLOWER
I20260619 14:32:37.613837 10108 consensus_queue.cc:260] T 722050258d2347f7b60552db8dae2fee P db3354f894ef40e3a566e882876af4f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db3354f894ef40e3a566e882876af4f4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37821 } } peers { permanent_uuid: "625146abfd2c4086badda901ef08c98c" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 40949 } } peers { permanent_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44701 } }
I20260619 14:32:37.614069 10108 ts_tablet_manager.cc:1428] T 722050258d2347f7b60552db8dae2fee P db3354f894ef40e3a566e882876af4f4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:32:37.614245 10110 tablet_bootstrap.cc:654] T 722050258d2347f7b60552db8dae2fee P d59bfc2b333a46bf94f4b782a7e388cd: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:37.614228 9584 catalog_manager.cc:4841] TS d59bfc2b333a46bf94f4b782a7e388cd (127.2.2.3:44701): tablet 6fa96f1330b942d68fd14dad9c69a527 (table test-table [id=00113256b29d42e2ba0a22c090e3c693]) successfully deleted
I20260619 14:32:37.614456 9874 tablet_service.cc:1467] Processing CreateTablet for tablet 722050258d2347f7b60552db8dae2fee (DEFAULT_TABLE table=test-table [id=ec5449b9956f4636a41f122ccbd18054]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:32:37.614530 9584 catalog_manager.cc:4841] TS db3354f894ef40e3a566e882876af4f4 (127.2.2.1:37821): tablet 6fa96f1330b942d68fd14dad9c69a527 (table test-table [id=00113256b29d42e2ba0a22c090e3c693]) successfully deleted
I20260619 14:32:37.614711 9874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 722050258d2347f7b60552db8dae2fee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:37.615834 10110 tablet_bootstrap.cc:492] T 722050258d2347f7b60552db8dae2fee P d59bfc2b333a46bf94f4b782a7e388cd: No bootstrap required, opened a new log
I20260619 14:32:37.615895 10110 ts_tablet_manager.cc:1397] T 722050258d2347f7b60552db8dae2fee P d59bfc2b333a46bf94f4b782a7e388cd: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260619 14:32:37.616343 10110 raft_consensus.cc:348] T 722050258d2347f7b60552db8dae2fee P d59bfc2b333a46bf94f4b782a7e388cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db3354f894ef40e3a566e882876af4f4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37821 } } peers { permanent_uuid: "625146abfd2c4086badda901ef08c98c" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 40949 } } peers { permanent_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44701 } }
I20260619 14:32:37.616442 10110 raft_consensus.cc:374] T 722050258d2347f7b60552db8dae2fee P d59bfc2b333a46bf94f4b782a7e388cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:37.616470 10110 raft_consensus.cc:729] T 722050258d2347f7b60552db8dae2fee P d59bfc2b333a46bf94f4b782a7e388cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d59bfc2b333a46bf94f4b782a7e388cd, State: Initialized, Role: FOLLOWER
I20260619 14:32:37.616611 10110 consensus_queue.cc:260] T 722050258d2347f7b60552db8dae2fee P d59bfc2b333a46bf94f4b782a7e388cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db3354f894ef40e3a566e882876af4f4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37821 } } peers { permanent_uuid: "625146abfd2c4086badda901ef08c98c" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 40949 } } peers { permanent_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44701 } }
I20260619 14:32:37.616870 10110 ts_tablet_manager.cc:1428] T 722050258d2347f7b60552db8dae2fee P d59bfc2b333a46bf94f4b782a7e388cd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 14:32:37.617197 10109 tablet_bootstrap.cc:492] T 722050258d2347f7b60552db8dae2fee P 625146abfd2c4086badda901ef08c98c: Bootstrap starting.
I20260619 14:32:37.618230 10109 tablet_bootstrap.cc:654] T 722050258d2347f7b60552db8dae2fee P 625146abfd2c4086badda901ef08c98c: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:37.619501 10109 tablet_bootstrap.cc:492] T 722050258d2347f7b60552db8dae2fee P 625146abfd2c4086badda901ef08c98c: No bootstrap required, opened a new log
I20260619 14:32:37.619614 10109 ts_tablet_manager.cc:1397] T 722050258d2347f7b60552db8dae2fee P 625146abfd2c4086badda901ef08c98c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260619 14:32:37.620036 10109 raft_consensus.cc:348] T 722050258d2347f7b60552db8dae2fee P 625146abfd2c4086badda901ef08c98c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db3354f894ef40e3a566e882876af4f4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37821 } } peers { permanent_uuid: "625146abfd2c4086badda901ef08c98c" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 40949 } } peers { permanent_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44701 } }
I20260619 14:32:37.620155 10109 raft_consensus.cc:374] T 722050258d2347f7b60552db8dae2fee P 625146abfd2c4086badda901ef08c98c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:37.620193 10109 raft_consensus.cc:729] T 722050258d2347f7b60552db8dae2fee P 625146abfd2c4086badda901ef08c98c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 625146abfd2c4086badda901ef08c98c, State: Initialized, Role: FOLLOWER
I20260619 14:32:37.620308 10109 consensus_queue.cc:260] T 722050258d2347f7b60552db8dae2fee P 625146abfd2c4086badda901ef08c98c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db3354f894ef40e3a566e882876af4f4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37821 } } peers { permanent_uuid: "625146abfd2c4086badda901ef08c98c" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 40949 } } peers { permanent_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44701 } }
I20260619 14:32:37.620586 10109 ts_tablet_manager.cc:1428] T 722050258d2347f7b60552db8dae2fee P 625146abfd2c4086badda901ef08c98c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
W20260619 14:32:37.628065 10081 tablet.cc:2369] T 722050258d2347f7b60552db8dae2fee P d59bfc2b333a46bf94f4b782a7e388cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:32:37.648599 9941 tablet.cc:2369] T 722050258d2347f7b60552db8dae2fee P 625146abfd2c4086badda901ef08c98c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:32:37.674181 9801 tablet.cc:2369] T 722050258d2347f7b60552db8dae2fee P db3354f894ef40e3a566e882876af4f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:32:37.698115 10118 raft_consensus.cc:482] T 722050258d2347f7b60552db8dae2fee P 625146abfd2c4086badda901ef08c98c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:32:37.698334 10118 raft_consensus.cc:504] T 722050258d2347f7b60552db8dae2fee P 625146abfd2c4086badda901ef08c98c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db3354f894ef40e3a566e882876af4f4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37821 } } peers { permanent_uuid: "625146abfd2c4086badda901ef08c98c" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 40949 } } peers { permanent_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44701 } }
I20260619 14:32:37.698791 10118 leader_election.cc:288] T 722050258d2347f7b60552db8dae2fee P 625146abfd2c4086badda901ef08c98c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db3354f894ef40e3a566e882876af4f4 (127.2.2.1:37821), d59bfc2b333a46bf94f4b782a7e388cd (127.2.2.3:44701)
I20260619 14:32:37.699271 9754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "722050258d2347f7b60552db8dae2fee" candidate_uuid: "625146abfd2c4086badda901ef08c98c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db3354f894ef40e3a566e882876af4f4" is_pre_election: true
I20260619 14:32:37.699271 10034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "722050258d2347f7b60552db8dae2fee" candidate_uuid: "625146abfd2c4086badda901ef08c98c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" is_pre_election: true
I20260619 14:32:37.699422 9754 raft_consensus.cc:2454] T 722050258d2347f7b60552db8dae2fee P db3354f894ef40e3a566e882876af4f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 625146abfd2c4086badda901ef08c98c in term 0.
I20260619 14:32:37.699422 10034 raft_consensus.cc:2454] T 722050258d2347f7b60552db8dae2fee P d59bfc2b333a46bf94f4b782a7e388cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 625146abfd2c4086badda901ef08c98c in term 0.
I20260619 14:32:37.699693 9828 leader_election.cc:302] T 722050258d2347f7b60552db8dae2fee P 625146abfd2c4086badda901ef08c98c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 625146abfd2c4086badda901ef08c98c, d59bfc2b333a46bf94f4b782a7e388cd; no voters:
I20260619 14:32:37.699898 10118 raft_consensus.cc:2781] T 722050258d2347f7b60552db8dae2fee P 625146abfd2c4086badda901ef08c98c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:32:37.699956 10118 raft_consensus.cc:482] T 722050258d2347f7b60552db8dae2fee P 625146abfd2c4086badda901ef08c98c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:32:37.700008 10118 raft_consensus.cc:3037] T 722050258d2347f7b60552db8dae2fee P 625146abfd2c4086badda901ef08c98c [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:37.700963 10118 raft_consensus.cc:504] T 722050258d2347f7b60552db8dae2fee P 625146abfd2c4086badda901ef08c98c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db3354f894ef40e3a566e882876af4f4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37821 } } peers { permanent_uuid: "625146abfd2c4086badda901ef08c98c" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 40949 } } peers { permanent_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44701 } }
I20260619 14:32:37.701330 10118 leader_election.cc:288] T 722050258d2347f7b60552db8dae2fee P 625146abfd2c4086badda901ef08c98c [CANDIDATE]: Term 1 election: Requested vote from peers db3354f894ef40e3a566e882876af4f4 (127.2.2.1:37821), d59bfc2b333a46bf94f4b782a7e388cd (127.2.2.3:44701)
I20260619 14:32:37.701712 9754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "722050258d2347f7b60552db8dae2fee" candidate_uuid: "625146abfd2c4086badda901ef08c98c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db3354f894ef40e3a566e882876af4f4"
I20260619 14:32:37.701712 10034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "722050258d2347f7b60552db8dae2fee" candidate_uuid: "625146abfd2c4086badda901ef08c98c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d59bfc2b333a46bf94f4b782a7e388cd"
I20260619 14:32:37.701835 10034 raft_consensus.cc:3037] T 722050258d2347f7b60552db8dae2fee P d59bfc2b333a46bf94f4b782a7e388cd [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:37.701835 9754 raft_consensus.cc:3037] T 722050258d2347f7b60552db8dae2fee P db3354f894ef40e3a566e882876af4f4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:37.702976 10034 raft_consensus.cc:2454] T 722050258d2347f7b60552db8dae2fee P d59bfc2b333a46bf94f4b782a7e388cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 625146abfd2c4086badda901ef08c98c in term 1.
I20260619 14:32:37.702976 9754 raft_consensus.cc:2454] T 722050258d2347f7b60552db8dae2fee P db3354f894ef40e3a566e882876af4f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 625146abfd2c4086badda901ef08c98c in term 1.
I20260619 14:32:37.703259 9828 leader_election.cc:302] T 722050258d2347f7b60552db8dae2fee P 625146abfd2c4086badda901ef08c98c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 625146abfd2c4086badda901ef08c98c, db3354f894ef40e3a566e882876af4f4; no voters:
I20260619 14:32:37.703456 10118 raft_consensus.cc:2781] T 722050258d2347f7b60552db8dae2fee P 625146abfd2c4086badda901ef08c98c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:32:37.703553 10118 raft_consensus.cc:686] T 722050258d2347f7b60552db8dae2fee P 625146abfd2c4086badda901ef08c98c [term 1 LEADER]: Becoming Leader. State: Replica: 625146abfd2c4086badda901ef08c98c, State: Running, Role: LEADER
I20260619 14:32:37.703692 10118 consensus_queue.cc:237] T 722050258d2347f7b60552db8dae2fee P 625146abfd2c4086badda901ef08c98c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db3354f894ef40e3a566e882876af4f4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37821 } } peers { permanent_uuid: "625146abfd2c4086badda901ef08c98c" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 40949 } } peers { permanent_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44701 } }
I20260619 14:32:37.705404 9596 catalog_manager.cc:5495] T 722050258d2347f7b60552db8dae2fee P 625146abfd2c4086badda901ef08c98c reported cstate change: term changed from 0 to 1, leader changed from <none> to 625146abfd2c4086badda901ef08c98c (127.2.2.2). New cstate: current_term: 1 leader_uuid: "625146abfd2c4086badda901ef08c98c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db3354f894ef40e3a566e882876af4f4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "625146abfd2c4086badda901ef08c98c" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 40949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44701 } health_report { overall_health: UNKNOWN } } }
Jun 19 14:32:37 dist-test-slave-70rm krb5kdc[9552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879556, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.2.62@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM
I20260619 14:32:37.741662 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:37.732054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33104 (local address 127.2.2.62:38669)
0619 14:32:37.732204 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:37.732209 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:37.732408 (+ 199us) server_negotiation.cc:407] Connection header received
0619 14:32:37.732554 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:37.732557 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:37.732597 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:37.732668 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:37.733578 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:37.734115 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:37.735306 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:37.735522 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:37.735611 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:37.738389 (+ 2778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:37.738392 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:37.738394 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:37.738415 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:37.739886 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:37.740392 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:37.740398 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:37.740400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:37.740475 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:37.740748 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:37.740753 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:37.740755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:37.740933 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:37.741037 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:37.741340 (+ 303us) server_negotiation.cc:299] Negotiation successful
0619 14:32:37.741502 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":68}
I20260619 14:32:37.741662 10144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:37.731962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:38669 (local address 127.0.0.1:33104)
0619 14:32:37.732297 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:37.732314 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:37.732453 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:37.732729 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:37.732732 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:37.733285 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:37.733419 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:37.733428 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:37.734294 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:37.734298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:37.735170 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:37.735181 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:37.735632 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:37.735656 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:37.735761 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:37.736441 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:37.736457 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:37.738204 (+ 1747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:37.740038 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:37.740049 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:37.740056 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:37.740295 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:37.740570 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:37.740573 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:37.740575 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:37.740663 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:37.741050 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:37.741059 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:37.741172 (+ 113us) client_negotiation.cc:769] Sending connection context
0619 14:32:37.741284 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 14:32:37.741447 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":121,"threads_started":1}
I20260619 14:32:37.746328 9596 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:33104:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 14:32:37.747248 9596 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:32:37.752767 9734 tablet_service.cc:1467] Processing CreateTablet for tablet fccae72e38fe4ca6912ab80a6a600a76 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=87b4a39e8ae347e3b5d03bd0eee4052c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 14:32:37.752767 9874 tablet_service.cc:1467] Processing CreateTablet for tablet fccae72e38fe4ca6912ab80a6a600a76 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=87b4a39e8ae347e3b5d03bd0eee4052c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 14:32:37.752949 10014 tablet_service.cc:1467] Processing CreateTablet for tablet fccae72e38fe4ca6912ab80a6a600a76 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=87b4a39e8ae347e3b5d03bd0eee4052c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 14:32:37.753180 9734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fccae72e38fe4ca6912ab80a6a600a76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:37.753180 9874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fccae72e38fe4ca6912ab80a6a600a76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:37.753322 10014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fccae72e38fe4ca6912ab80a6a600a76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:37.756067 10108 tablet_bootstrap.cc:492] T fccae72e38fe4ca6912ab80a6a600a76 P db3354f894ef40e3a566e882876af4f4: Bootstrap starting.
I20260619 14:32:37.756062 10109 tablet_bootstrap.cc:492] T fccae72e38fe4ca6912ab80a6a600a76 P 625146abfd2c4086badda901ef08c98c: Bootstrap starting.
I20260619 14:32:37.756484 10110 tablet_bootstrap.cc:492] T fccae72e38fe4ca6912ab80a6a600a76 P d59bfc2b333a46bf94f4b782a7e388cd: Bootstrap starting.
I20260619 14:32:37.757012 10109 tablet_bootstrap.cc:654] T fccae72e38fe4ca6912ab80a6a600a76 P 625146abfd2c4086badda901ef08c98c: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:37.757012 10108 tablet_bootstrap.cc:654] T fccae72e38fe4ca6912ab80a6a600a76 P db3354f894ef40e3a566e882876af4f4: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:37.757527 10110 tablet_bootstrap.cc:654] T fccae72e38fe4ca6912ab80a6a600a76 P d59bfc2b333a46bf94f4b782a7e388cd: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:37.758715 10108 tablet_bootstrap.cc:492] T fccae72e38fe4ca6912ab80a6a600a76 P db3354f894ef40e3a566e882876af4f4: No bootstrap required, opened a new log
I20260619 14:32:37.758809 10108 ts_tablet_manager.cc:1397] T fccae72e38fe4ca6912ab80a6a600a76 P db3354f894ef40e3a566e882876af4f4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 14:32:37.759083 10110 tablet_bootstrap.cc:492] T fccae72e38fe4ca6912ab80a6a600a76 P d59bfc2b333a46bf94f4b782a7e388cd: No bootstrap required, opened a new log
I20260619 14:32:37.759162 10110 ts_tablet_manager.cc:1397] T fccae72e38fe4ca6912ab80a6a600a76 P d59bfc2b333a46bf94f4b782a7e388cd: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260619 14:32:37.759186 10108 raft_consensus.cc:348] T fccae72e38fe4ca6912ab80a6a600a76 P db3354f894ef40e3a566e882876af4f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db3354f894ef40e3a566e882876af4f4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37821 } } peers { permanent_uuid: "625146abfd2c4086badda901ef08c98c" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 40949 } } peers { permanent_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44701 } }
I20260619 14:32:37.759276 10108 raft_consensus.cc:374] T fccae72e38fe4ca6912ab80a6a600a76 P db3354f894ef40e3a566e882876af4f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:37.759296 10108 raft_consensus.cc:729] T fccae72e38fe4ca6912ab80a6a600a76 P db3354f894ef40e3a566e882876af4f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db3354f894ef40e3a566e882876af4f4, State: Initialized, Role: FOLLOWER
I20260619 14:32:37.759444 10108 consensus_queue.cc:260] T fccae72e38fe4ca6912ab80a6a600a76 P db3354f894ef40e3a566e882876af4f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db3354f894ef40e3a566e882876af4f4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37821 } } peers { permanent_uuid: "625146abfd2c4086badda901ef08c98c" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 40949 } } peers { permanent_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44701 } }
I20260619 14:32:37.759574 10110 raft_consensus.cc:348] T fccae72e38fe4ca6912ab80a6a600a76 P d59bfc2b333a46bf94f4b782a7e388cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db3354f894ef40e3a566e882876af4f4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37821 } } peers { permanent_uuid: "625146abfd2c4086badda901ef08c98c" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 40949 } } peers { permanent_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44701 } }
I20260619 14:32:37.759682 10110 raft_consensus.cc:374] T fccae72e38fe4ca6912ab80a6a600a76 P d59bfc2b333a46bf94f4b782a7e388cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:37.759711 10110 raft_consensus.cc:729] T fccae72e38fe4ca6912ab80a6a600a76 P d59bfc2b333a46bf94f4b782a7e388cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d59bfc2b333a46bf94f4b782a7e388cd, State: Initialized, Role: FOLLOWER
I20260619 14:32:37.759764 10108 ts_tablet_manager.cc:1428] T fccae72e38fe4ca6912ab80a6a600a76 P db3354f894ef40e3a566e882876af4f4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 14:32:37.759706 10114 tablet_replica.cc:439] T fccae72e38fe4ca6912ab80a6a600a76 P db3354f894ef40e3a566e882876af4f4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db3354f894ef40e3a566e882876af4f4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37821 } } peers { permanent_uuid: "625146abfd2c4086badda901ef08c98c" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 40949 } } peers { permanent_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44701 } } }
I20260619 14:32:37.759840 10114 tablet_replica.cc:442] T fccae72e38fe4ca6912ab80a6a600a76 P db3354f894ef40e3a566e882876af4f4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:32:37.759840 10110 consensus_queue.cc:260] T fccae72e38fe4ca6912ab80a6a600a76 P d59bfc2b333a46bf94f4b782a7e388cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db3354f894ef40e3a566e882876af4f4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37821 } } peers { permanent_uuid: "625146abfd2c4086badda901ef08c98c" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 40949 } } peers { permanent_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44701 } }
I20260619 14:32:37.760152 10110 ts_tablet_manager.cc:1428] T fccae72e38fe4ca6912ab80a6a600a76 P d59bfc2b333a46bf94f4b782a7e388cd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:32:37.760572 10109 tablet_bootstrap.cc:492] T fccae72e38fe4ca6912ab80a6a600a76 P 625146abfd2c4086badda901ef08c98c: No bootstrap required, opened a new log
I20260619 14:32:37.760667 10109 ts_tablet_manager.cc:1397] T fccae72e38fe4ca6912ab80a6a600a76 P 625146abfd2c4086badda901ef08c98c: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260619 14:32:37.761000 10109 raft_consensus.cc:348] T fccae72e38fe4ca6912ab80a6a600a76 P 625146abfd2c4086badda901ef08c98c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db3354f894ef40e3a566e882876af4f4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37821 } } peers { permanent_uuid: "625146abfd2c4086badda901ef08c98c" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 40949 } } peers { permanent_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44701 } }
I20260619 14:32:37.761085 10109 raft_consensus.cc:374] T fccae72e38fe4ca6912ab80a6a600a76 P 625146abfd2c4086badda901ef08c98c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:37.761106 10109 raft_consensus.cc:729] T fccae72e38fe4ca6912ab80a6a600a76 P 625146abfd2c4086badda901ef08c98c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 625146abfd2c4086badda901ef08c98c, State: Initialized, Role: FOLLOWER
I20260619 14:32:37.761238 10109 consensus_queue.cc:260] T fccae72e38fe4ca6912ab80a6a600a76 P 625146abfd2c4086badda901ef08c98c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db3354f894ef40e3a566e882876af4f4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37821 } } peers { permanent_uuid: "625146abfd2c4086badda901ef08c98c" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 40949 } } peers { permanent_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44701 } }
I20260619 14:32:37.761519 10109 ts_tablet_manager.cc:1428] T fccae72e38fe4ca6912ab80a6a600a76 P 625146abfd2c4086badda901ef08c98c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 14:32:37.761466 10118 tablet_replica.cc:439] T fccae72e38fe4ca6912ab80a6a600a76 P 625146abfd2c4086badda901ef08c98c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db3354f894ef40e3a566e882876af4f4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37821 } } peers { permanent_uuid: "625146abfd2c4086badda901ef08c98c" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 40949 } } peers { permanent_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44701 } } }
I20260619 14:32:37.761592 10118 tablet_replica.cc:442] T fccae72e38fe4ca6912ab80a6a600a76 P 625146abfd2c4086badda901ef08c98c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:32:37.763310 10116 tablet_replica.cc:439] T fccae72e38fe4ca6912ab80a6a600a76 P d59bfc2b333a46bf94f4b782a7e388cd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db3354f894ef40e3a566e882876af4f4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37821 } } peers { permanent_uuid: "625146abfd2c4086badda901ef08c98c" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 40949 } } peers { permanent_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44701 } } }
I20260619 14:32:37.763535 10116 tablet_replica.cc:442] T fccae72e38fe4ca6912ab80a6a600a76 P d59bfc2b333a46bf94f4b782a7e388cd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:32:37.857800 10114 raft_consensus.cc:482] T fccae72e38fe4ca6912ab80a6a600a76 P db3354f894ef40e3a566e882876af4f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:32:37.857980 10114 raft_consensus.cc:504] T fccae72e38fe4ca6912ab80a6a600a76 P db3354f894ef40e3a566e882876af4f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db3354f894ef40e3a566e882876af4f4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37821 } } peers { permanent_uuid: "625146abfd2c4086badda901ef08c98c" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 40949 } } peers { permanent_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44701 } }
I20260619 14:32:37.859117 10114 leader_election.cc:288] T fccae72e38fe4ca6912ab80a6a600a76 P db3354f894ef40e3a566e882876af4f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 625146abfd2c4086badda901ef08c98c (127.2.2.2:40949), d59bfc2b333a46bf94f4b782a7e388cd (127.2.2.3:44701)
I20260619 14:32:37.863443 10106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:37.859493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:54959 (local address 127.2.2.3:44701)
0619 14:32:37.859752 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:37.859757 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:37.859776 (+ 19us) server_negotiation.cc:407] Connection header received
0619 14:32:37.859979 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:37.859984 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:37.860040 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:37.860100 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:32:37.860430 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:37.861061 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:37.862456 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:37.862861 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:37.862941 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:37.863033 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:37.863225 (+ 192us) server_negotiation.cc:299] Negotiation successful
0619 14:32:37.863303 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165}
I20260619 14:32:37.863456 10103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:37.859588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:54031 (local address 127.2.2.2:40949)
0619 14:32:37.859767 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:37.859773 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:37.859883 (+ 110us) server_negotiation.cc:407] Connection header received
0619 14:32:37.860028 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:37.860030 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:37.860067 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:37.860158 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:32:37.860475 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:37.861043 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:37.862482 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:37.862882 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:37.863005 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:37.863080 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:37.863275 (+ 195us) server_negotiation.cc:299] Negotiation successful
0619 14:32:37.863353 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 14:32:37.863476 10149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:37.859418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.2:40949 (local address 127.2.2.1:54031)
0619 14:32:37.859778 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:37.859793 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:37.859894 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:37.860189 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:37.860195 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:37.860211 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:32:37.860344 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:37.860349 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:37.861152 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:37.861158 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:37.862342 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:37.862351 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:37.863027 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:37.863030 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:37.863128 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:37.863144 (+ 16us) client_negotiation.cc:769] Sending connection context
0619 14:32:37.863245 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 14:32:37.863307 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":83,"thread_start_us":99,"threads_started":1}
I20260619 14:32:37.863482 10148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:37.859380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.3:44701 (local address 127.2.2.1:54959)
0619 14:32:37.859698 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:37.859714 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:37.859834 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:37.860135 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:37.860140 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:37.860160 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:32:37.860321 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:37.860327 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:37.861190 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:37.861196 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:37.862330 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:37.862339 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:37.862978 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:37.862981 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:37.863082 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:37.863114 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 14:32:37.863182 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 14:32:37.863273 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":65,"thread_start_us":129,"threads_started":1}
I20260619 14:32:37.864138 9894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fccae72e38fe4ca6912ab80a6a600a76" candidate_uuid: "db3354f894ef40e3a566e882876af4f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "625146abfd2c4086badda901ef08c98c" is_pre_election: true
I20260619 14:32:37.864375 10034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fccae72e38fe4ca6912ab80a6a600a76" candidate_uuid: "db3354f894ef40e3a566e882876af4f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" is_pre_election: true
I20260619 14:32:37.864392 9894 raft_consensus.cc:2454] T fccae72e38fe4ca6912ab80a6a600a76 P 625146abfd2c4086badda901ef08c98c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db3354f894ef40e3a566e882876af4f4 in term 0.
I20260619 14:32:37.864482 10034 raft_consensus.cc:2454] T fccae72e38fe4ca6912ab80a6a600a76 P d59bfc2b333a46bf94f4b782a7e388cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db3354f894ef40e3a566e882876af4f4 in term 0.
I20260619 14:32:37.864840 9686 leader_election.cc:302] T fccae72e38fe4ca6912ab80a6a600a76 P db3354f894ef40e3a566e882876af4f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 625146abfd2c4086badda901ef08c98c, d59bfc2b333a46bf94f4b782a7e388cd, db3354f894ef40e3a566e882876af4f4; no voters:
I20260619 14:32:37.865106 10114 raft_consensus.cc:2781] T fccae72e38fe4ca6912ab80a6a600a76 P db3354f894ef40e3a566e882876af4f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:32:37.865177 10114 raft_consensus.cc:482] T fccae72e38fe4ca6912ab80a6a600a76 P db3354f894ef40e3a566e882876af4f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:32:37.865231 10114 raft_consensus.cc:3037] T fccae72e38fe4ca6912ab80a6a600a76 P db3354f894ef40e3a566e882876af4f4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:37.866220 10114 raft_consensus.cc:504] T fccae72e38fe4ca6912ab80a6a600a76 P db3354f894ef40e3a566e882876af4f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db3354f894ef40e3a566e882876af4f4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37821 } } peers { permanent_uuid: "625146abfd2c4086badda901ef08c98c" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 40949 } } peers { permanent_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44701 } }
I20260619 14:32:37.866627 10114 leader_election.cc:288] T fccae72e38fe4ca6912ab80a6a600a76 P db3354f894ef40e3a566e882876af4f4 [CANDIDATE]: Term 1 election: Requested vote from peers 625146abfd2c4086badda901ef08c98c (127.2.2.2:40949), d59bfc2b333a46bf94f4b782a7e388cd (127.2.2.3:44701)
I20260619 14:32:37.867074 9894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fccae72e38fe4ca6912ab80a6a600a76" candidate_uuid: "db3354f894ef40e3a566e882876af4f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "625146abfd2c4086badda901ef08c98c"
I20260619 14:32:37.867090 10034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fccae72e38fe4ca6912ab80a6a600a76" candidate_uuid: "db3354f894ef40e3a566e882876af4f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d59bfc2b333a46bf94f4b782a7e388cd"
I20260619 14:32:37.867210 10034 raft_consensus.cc:3037] T fccae72e38fe4ca6912ab80a6a600a76 P d59bfc2b333a46bf94f4b782a7e388cd [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:37.867187 9894 raft_consensus.cc:3037] T fccae72e38fe4ca6912ab80a6a600a76 P 625146abfd2c4086badda901ef08c98c [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:37.868093 10034 raft_consensus.cc:2454] T fccae72e38fe4ca6912ab80a6a600a76 P d59bfc2b333a46bf94f4b782a7e388cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db3354f894ef40e3a566e882876af4f4 in term 1.
I20260619 14:32:37.868227 9894 raft_consensus.cc:2454] T fccae72e38fe4ca6912ab80a6a600a76 P 625146abfd2c4086badda901ef08c98c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db3354f894ef40e3a566e882876af4f4 in term 1.
I20260619 14:32:37.868459 9688 leader_election.cc:302] T fccae72e38fe4ca6912ab80a6a600a76 P db3354f894ef40e3a566e882876af4f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d59bfc2b333a46bf94f4b782a7e388cd, db3354f894ef40e3a566e882876af4f4; no voters:
I20260619 14:32:37.868729 10114 raft_consensus.cc:2781] T fccae72e38fe4ca6912ab80a6a600a76 P db3354f894ef40e3a566e882876af4f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:32:37.868983 10114 raft_consensus.cc:686] T fccae72e38fe4ca6912ab80a6a600a76 P db3354f894ef40e3a566e882876af4f4 [term 1 LEADER]: Becoming Leader. State: Replica: db3354f894ef40e3a566e882876af4f4, State: Running, Role: LEADER
I20260619 14:32:37.869284 10114 consensus_queue.cc:237] T fccae72e38fe4ca6912ab80a6a600a76 P db3354f894ef40e3a566e882876af4f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db3354f894ef40e3a566e882876af4f4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37821 } } peers { permanent_uuid: "625146abfd2c4086badda901ef08c98c" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 40949 } } peers { permanent_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44701 } }
I20260619 14:32:37.870604 10150 tablet_replica.cc:439] T fccae72e38fe4ca6912ab80a6a600a76 P db3354f894ef40e3a566e882876af4f4: TxnStatusTablet state changed. Reason: New leader db3354f894ef40e3a566e882876af4f4. Latest consensus state: current_term: 1 leader_uuid: "db3354f894ef40e3a566e882876af4f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db3354f894ef40e3a566e882876af4f4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37821 } } peers { permanent_uuid: "625146abfd2c4086badda901ef08c98c" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 40949 } } peers { permanent_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44701 } } }
I20260619 14:32:37.870724 10150 tablet_replica.cc:442] T fccae72e38fe4ca6912ab80a6a600a76 P db3354f894ef40e3a566e882876af4f4: This TxnStatusTablet replica's current role is: LEADER
I20260619 14:32:37.871018 10152 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 14:32:37.871177 10152 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 14:32:37.871894 9597 catalog_manager.cc:5495] T fccae72e38fe4ca6912ab80a6a600a76 P db3354f894ef40e3a566e882876af4f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to db3354f894ef40e3a566e882876af4f4 (127.2.2.1). New cstate: current_term: 1 leader_uuid: "db3354f894ef40e3a566e882876af4f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db3354f894ef40e3a566e882876af4f4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "625146abfd2c4086badda901ef08c98c" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 40949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44701 } health_report { overall_health: UNKNOWN } } }
I20260619 14:32:37.895247 10107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:37.891177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42506 (local address 127.2.2.1:37821)
0619 14:32:37.891376 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:37.891386 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:37.891402 (+ 16us) server_negotiation.cc:407] Connection header received
0619 14:32:37.891597 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:37.891600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:37.891654 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:37.891739 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:32:37.892038 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:37.892539 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:37.893763 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:37.893979 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:37.894059 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:37.894440 (+ 381us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:32:37.894866 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:32:37.894950 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:37.895061 (+ 111us) server_negotiation.cc:299] Negotiation successful
0619 14:32:37.895112 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 14:32:37.895296 10144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:37.891034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.1:37821 (local address 127.0.0.1:42506)
0619 14:32:37.891334 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:37.891350 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:37.891456 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:37.891796 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:37.891799 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:37.891809 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:32:37.891929 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:37.891936 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:37.892643 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:37.892646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:37.893647 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:37.893658 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:37.894079 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:37.894095 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:37.894228 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:37.894285 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:32:37.894963 (+ 678us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:32:37.894977 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 14:32:37.895053 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 14:32:37.895143 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179}
I20260619 14:32:37.898491 10034 raft_consensus.cc:1261] T fccae72e38fe4ca6912ab80a6a600a76 P d59bfc2b333a46bf94f4b782a7e388cd [term 1 FOLLOWER]: Refusing update from remote peer db3354f894ef40e3a566e882876af4f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:32:37.898649 9894 raft_consensus.cc:1261] T fccae72e38fe4ca6912ab80a6a600a76 P 625146abfd2c4086badda901ef08c98c [term 1 FOLLOWER]: Refusing update from remote peer db3354f894ef40e3a566e882876af4f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:32:37.899044 10114 consensus_queue.cc:1035] T fccae72e38fe4ca6912ab80a6a600a76 P db3354f894ef40e3a566e882876af4f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:32:37.899204 10150 consensus_queue.cc:1035] T fccae72e38fe4ca6912ab80a6a600a76 P db3354f894ef40e3a566e882876af4f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "625146abfd2c4086badda901ef08c98c" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 40949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:32:37.900682 10116 tablet_replica.cc:439] T fccae72e38fe4ca6912ab80a6a600a76 P d59bfc2b333a46bf94f4b782a7e388cd: TxnStatusTablet state changed. Reason: New leader db3354f894ef40e3a566e882876af4f4. Latest consensus state: current_term: 1 leader_uuid: "db3354f894ef40e3a566e882876af4f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db3354f894ef40e3a566e882876af4f4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37821 } } peers { permanent_uuid: "625146abfd2c4086badda901ef08c98c" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 40949 } } peers { permanent_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44701 } } }
I20260619 14:32:37.900810 10116 tablet_replica.cc:442] T fccae72e38fe4ca6912ab80a6a600a76 P d59bfc2b333a46bf94f4b782a7e388cd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:32:37.900928 10118 tablet_replica.cc:439] T fccae72e38fe4ca6912ab80a6a600a76 P 625146abfd2c4086badda901ef08c98c: TxnStatusTablet state changed. Reason: New leader db3354f894ef40e3a566e882876af4f4. Latest consensus state: current_term: 1 leader_uuid: "db3354f894ef40e3a566e882876af4f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db3354f894ef40e3a566e882876af4f4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37821 } } peers { permanent_uuid: "625146abfd2c4086badda901ef08c98c" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 40949 } } peers { permanent_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44701 } } }
I20260619 14:32:37.901041 10118 tablet_replica.cc:442] T fccae72e38fe4ca6912ab80a6a600a76 P 625146abfd2c4086badda901ef08c98c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:32:37.902768 10114 tablet_replica.cc:439] T fccae72e38fe4ca6912ab80a6a600a76 P db3354f894ef40e3a566e882876af4f4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "db3354f894ef40e3a566e882876af4f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db3354f894ef40e3a566e882876af4f4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37821 } } peers { permanent_uuid: "625146abfd2c4086badda901ef08c98c" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 40949 } } peers { permanent_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44701 } } }
I20260619 14:32:37.902904 10114 tablet_replica.cc:442] T fccae72e38fe4ca6912ab80a6a600a76 P db3354f894ef40e3a566e882876af4f4: This TxnStatusTablet replica's current role is: LEADER
I20260619 14:32:37.903409 10116 tablet_replica.cc:439] T fccae72e38fe4ca6912ab80a6a600a76 P d59bfc2b333a46bf94f4b782a7e388cd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "db3354f894ef40e3a566e882876af4f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db3354f894ef40e3a566e882876af4f4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37821 } } peers { permanent_uuid: "625146abfd2c4086badda901ef08c98c" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 40949 } } peers { permanent_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44701 } } }
I20260619 14:32:37.903515 10116 tablet_replica.cc:442] T fccae72e38fe4ca6912ab80a6a600a76 P d59bfc2b333a46bf94f4b782a7e388cd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:32:37.905478 10114 tablet_replica.cc:439] T fccae72e38fe4ca6912ab80a6a600a76 P db3354f894ef40e3a566e882876af4f4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "db3354f894ef40e3a566e882876af4f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db3354f894ef40e3a566e882876af4f4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37821 } } peers { permanent_uuid: "625146abfd2c4086badda901ef08c98c" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 40949 } } peers { permanent_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44701 } } }
I20260619 14:32:37.905596 10114 tablet_replica.cc:442] T fccae72e38fe4ca6912ab80a6a600a76 P db3354f894ef40e3a566e882876af4f4: This TxnStatusTablet replica's current role is: LEADER
I20260619 14:32:37.906505 10121 mvcc.cc:205] Tried to move back new op lower bound from 7298578669146980352 to 7298578669035048960. Current Snapshot: MvccSnapshot[applied={T|T < 7298578669146980352}]
I20260619 14:32:37.906652 10118 tablet_replica.cc:439] T fccae72e38fe4ca6912ab80a6a600a76 P 625146abfd2c4086badda901ef08c98c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "db3354f894ef40e3a566e882876af4f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db3354f894ef40e3a566e882876af4f4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37821 } } peers { permanent_uuid: "625146abfd2c4086badda901ef08c98c" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 40949 } } peers { permanent_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44701 } } }
I20260619 14:32:37.906780 10118 tablet_replica.cc:442] T fccae72e38fe4ca6912ab80a6a600a76 P 625146abfd2c4086badda901ef08c98c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:32:37.919139 10107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:37.915318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:43677 (local address 127.2.2.1:37821)
0619 14:32:37.915460 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:37.915465 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:37.915671 (+ 206us) server_negotiation.cc:407] Connection header received
0619 14:32:37.915807 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:37.915809 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:37.915864 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:37.915928 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:32:37.916192 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:37.916679 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:37.917958 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:37.918274 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:37.918353 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:37.918668 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:32:37.918782 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:32:37.918866 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:37.918980 (+ 114us) server_negotiation.cc:299] Negotiation successful
0619 14:32:37.919026 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 14:32:37.919211 10159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:37.915227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.1:37821 (local address 127.2.2.2:43677)
0619 14:32:37.915580 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:37.915596 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:37.915713 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:37.915959 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:37.915961 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:37.915970 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:32:37.916092 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:37.916098 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:37.916781 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:37.916784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:37.917840 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:37.917856 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:37.918368 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:37.918387 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:37.918480 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:37.918537 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:32:37.918881 (+ 344us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:32:37.918888 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 14:32:37.918955 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 14:32:37.919044 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":129,"threads_started":1}
I20260619 14:32:37.926695 10034 raft_consensus.cc:1261] T 722050258d2347f7b60552db8dae2fee P d59bfc2b333a46bf94f4b782a7e388cd [term 1 FOLLOWER]: Refusing update from remote peer 625146abfd2c4086badda901ef08c98c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:32:37.926695 9754 raft_consensus.cc:1261] T 722050258d2347f7b60552db8dae2fee P db3354f894ef40e3a566e882876af4f4 [term 1 FOLLOWER]: Refusing update from remote peer 625146abfd2c4086badda901ef08c98c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:32:37.927131 10115 consensus_queue.cc:1035] T 722050258d2347f7b60552db8dae2fee P 625146abfd2c4086badda901ef08c98c [LEADER]: Connected to new peer: Peer: permanent_uuid: "db3354f894ef40e3a566e882876af4f4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:32:37.927280 10118 consensus_queue.cc:1035] T 722050258d2347f7b60552db8dae2fee P 625146abfd2c4086badda901ef08c98c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:32:37.954586 10164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:37.948372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.2:40949 (local address 127.2.2.1:55899)
0619 14:32:37.950398 (+ 2026us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:37.950418 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:37.950578 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:37.950949 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:37.950951 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:37.950959 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:32:37.951092 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:37.951098 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:37.952154 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:37.952157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:37.953175 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:37.953186 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:37.953723 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:37.953739 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:37.953851 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:37.953912 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:32:37.954256 (+ 344us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:32:37.954264 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 14:32:37.954335 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 14:32:37.954421 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1896,"thread_start_us":98,"threads_started":1}
I20260619 14:32:37.954651 10103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:37.948711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:55899 (local address 127.2.2.2:40949)
0619 14:32:37.948834 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:37.948839 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:37.950500 (+ 1661us) server_negotiation.cc:407] Connection header received
0619 14:32:37.950695 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:37.950698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:37.950753 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:37.950816 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:32:37.951252 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:37.952021 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:37.953311 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:37.953615 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:37.953723 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:37.954044 (+ 321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:32:37.954158 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:32:37.954231 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:37.954445 (+ 214us) server_negotiation.cc:299] Negotiation successful
0619 14:32:37.954506 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 14:32:37.976064 9597 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:32:37.976228 9597 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:32:37.978045 9597 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 50728c18eca74389b5c824d5d291fa09: Sending DeleteTablet for 3 replicas of tablet 722050258d2347f7b60552db8dae2fee
I20260619 14:32:37.978734 10014 tablet_service.cc:1514] Processing DeleteTablet for tablet 722050258d2347f7b60552db8dae2fee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:32:37 UTC) from {username='oryx', principal='oryx/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:50450
I20260619 14:32:37.978912 10132 tablet_replica.cc:330] T 722050258d2347f7b60552db8dae2fee P d59bfc2b333a46bf94f4b782a7e388cd: stopping tablet replica
I20260619 14:32:37.979028 10132 raft_consensus.cc:2229] T 722050258d2347f7b60552db8dae2fee P d59bfc2b333a46bf94f4b782a7e388cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:32:37.979141 10132 raft_consensus.cc:2258] T 722050258d2347f7b60552db8dae2fee P d59bfc2b333a46bf94f4b782a7e388cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:32:37.980659 9874 tablet_service.cc:1514] Processing DeleteTablet for tablet 722050258d2347f7b60552db8dae2fee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:32:37 UTC) from {username='oryx', principal='oryx/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:50778
I20260619 14:32:37.980840 10134 tablet_replica.cc:330] T 722050258d2347f7b60552db8dae2fee P 625146abfd2c4086badda901ef08c98c: stopping tablet replica
I20260619 14:32:37.980945 10134 raft_consensus.cc:2229] T 722050258d2347f7b60552db8dae2fee P 625146abfd2c4086badda901ef08c98c [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:32:37.981132 10134 raft_consensus.cc:2258] T 722050258d2347f7b60552db8dae2fee P 625146abfd2c4086badda901ef08c98c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:32:37.981454 10132 ts_tablet_manager.cc:1905] T 722050258d2347f7b60552db8dae2fee P d59bfc2b333a46bf94f4b782a7e388cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:32:37.981842 10134 ts_tablet_manager.cc:1905] T 722050258d2347f7b60552db8dae2fee P 625146abfd2c4086badda901ef08c98c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:32:37.984401 10132 ts_tablet_manager.cc:1918] T 722050258d2347f7b60552db8dae2fee P d59bfc2b333a46bf94f4b782a7e388cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:32:37.984401 10134 ts_tablet_manager.cc:1918] T 722050258d2347f7b60552db8dae2fee P 625146abfd2c4086badda901ef08c98c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:32:37.984485 10134 log.cc:1196] T 722050258d2347f7b60552db8dae2fee P 625146abfd2c4086badda901ef08c98c: Deleting WAL directory at /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/ts-1/wal/wals/722050258d2347f7b60552db8dae2fee
I20260619 14:32:37.984485 10132 log.cc:1196] T 722050258d2347f7b60552db8dae2fee P d59bfc2b333a46bf94f4b782a7e388cd: Deleting WAL directory at /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/ts-2/wal/wals/722050258d2347f7b60552db8dae2fee
I20260619 14:32:37.984736 10132 ts_tablet_manager.cc:1939] T 722050258d2347f7b60552db8dae2fee P d59bfc2b333a46bf94f4b782a7e388cd: Deleting consensus metadata
I20260619 14:32:37.984747 10134 ts_tablet_manager.cc:1939] T 722050258d2347f7b60552db8dae2fee P 625146abfd2c4086badda901ef08c98c: Deleting consensus metadata
I20260619 14:32:37.985407 9584 catalog_manager.cc:4841] TS d59bfc2b333a46bf94f4b782a7e388cd (127.2.2.3:44701): tablet 722050258d2347f7b60552db8dae2fee (table test-table [id=ec5449b9956f4636a41f122ccbd18054]) successfully deleted
I20260619 14:32:37.985457 9582 catalog_manager.cc:4841] TS 625146abfd2c4086badda901ef08c98c (127.2.2.2:40949): tablet 722050258d2347f7b60552db8dae2fee (table test-table [id=ec5449b9956f4636a41f122ccbd18054]) successfully deleted
I20260619 14:32:37.978636 9734 tablet_service.cc:1514] Processing DeleteTablet for tablet 722050258d2347f7b60552db8dae2fee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:32:37 UTC) from {username='oryx', principal='oryx/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:42494
I20260619 14:32:37.985793 10133 tablet_replica.cc:330] T 722050258d2347f7b60552db8dae2fee P db3354f894ef40e3a566e882876af4f4: stopping tablet replica
I20260619 14:32:37.985916 10133 raft_consensus.cc:2229] T 722050258d2347f7b60552db8dae2fee P db3354f894ef40e3a566e882876af4f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:32:37.986035 10133 raft_consensus.cc:2258] T 722050258d2347f7b60552db8dae2fee P db3354f894ef40e3a566e882876af4f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:32:37.986855 10133 ts_tablet_manager.cc:1905] T 722050258d2347f7b60552db8dae2fee P db3354f894ef40e3a566e882876af4f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:32:37.990257 10133 ts_tablet_manager.cc:1918] T 722050258d2347f7b60552db8dae2fee P db3354f894ef40e3a566e882876af4f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:32:37.990350 10133 log.cc:1196] T 722050258d2347f7b60552db8dae2fee P db3354f894ef40e3a566e882876af4f4: Deleting WAL directory at /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/ts-0/wal/wals/722050258d2347f7b60552db8dae2fee
I20260619 14:32:37.990648 10133 ts_tablet_manager.cc:1939] T 722050258d2347f7b60552db8dae2fee P db3354f894ef40e3a566e882876af4f4: Deleting consensus metadata
I20260619 14:32:37.991617 9584 catalog_manager.cc:4841] TS db3354f894ef40e3a566e882876af4f4 (127.2.2.1:37821): tablet 722050258d2347f7b60552db8dae2fee (table test-table [id=ec5449b9956f4636a41f122ccbd18054]) successfully deleted
Jun 19 14:32:37 dist-test-slave-70rm krb5kdc[9552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 14:32:38 dist-test-slave-70rm krb5kdc[9552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:32:38 dist-test-slave-70rm krb5kdc[9552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 14:32:38.012053 9646 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:32:38.005542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33108 (local address 127.2.2.62:38669)
0619 14:32:38.005695 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:38.005700 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:38.005874 (+ 174us) server_negotiation.cc:407] Connection header received
0619 14:32:38.006041 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:38.006044 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:38.006097 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:38.006215 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:38.006986 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:38.007531 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:38.008398 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:38.008624 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:38.008739 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:38.011907 (+ 3168us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33108: BlockingRecv error: recv got EOF from 127.0.0.1:33108 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 19 14:32:38 dist-test-slave-70rm krb5kdc[9552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM
I20260619 14:32:38.022586 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:38.013822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33116 (local address 127.2.2.62:38669)
0619 14:32:38.013966 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:38.013970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:38.014062 (+ 92us) server_negotiation.cc:407] Connection header received
0619 14:32:38.014290 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:38.014293 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:38.014344 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:38.014428 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:38.015026 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:38.015543 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:38.016348 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:38.016641 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:38.016751 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:38.019364 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:38.019368 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:38.019371 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:38.019406 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:38.020908 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:38.021341 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:38.021345 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:38.021347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:38.021428 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:38.021709 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:38.021713 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:38.021714 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:38.021906 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:38.022046 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:38.022292 (+ 246us) server_negotiation.cc:299] Negotiation successful
0619 14:32:38.022406 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 14:32:38.025918 9597 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33116:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:32:38.026489 9597 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:32:38.031294 9734 tablet_service.cc:1467] Processing CreateTablet for tablet 8ac654811cc549d8ad2179b11413097d (DEFAULT_TABLE table=test-table [id=953c94a3ddf7490d8cb3e3b201f80b10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:32:38.031359 9874 tablet_service.cc:1467] Processing CreateTablet for tablet 8ac654811cc549d8ad2179b11413097d (DEFAULT_TABLE table=test-table [id=953c94a3ddf7490d8cb3e3b201f80b10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:32:38.031620 9734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ac654811cc549d8ad2179b11413097d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:38.031672 9874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ac654811cc549d8ad2179b11413097d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:38.033540 10108 tablet_bootstrap.cc:492] T 8ac654811cc549d8ad2179b11413097d P db3354f894ef40e3a566e882876af4f4: Bootstrap starting.
I20260619 14:32:38.034006 10109 tablet_bootstrap.cc:492] T 8ac654811cc549d8ad2179b11413097d P 625146abfd2c4086badda901ef08c98c: Bootstrap starting.
I20260619 14:32:38.033967 10014 tablet_service.cc:1467] Processing CreateTablet for tablet 8ac654811cc549d8ad2179b11413097d (DEFAULT_TABLE table=test-table [id=953c94a3ddf7490d8cb3e3b201f80b10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:32:38.034512 10108 tablet_bootstrap.cc:654] T 8ac654811cc549d8ad2179b11413097d P db3354f894ef40e3a566e882876af4f4: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:38.034575 10014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ac654811cc549d8ad2179b11413097d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:38.034761 10109 tablet_bootstrap.cc:654] T 8ac654811cc549d8ad2179b11413097d P 625146abfd2c4086badda901ef08c98c: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:38.036552 10109 tablet_bootstrap.cc:492] T 8ac654811cc549d8ad2179b11413097d P 625146abfd2c4086badda901ef08c98c: No bootstrap required, opened a new log
I20260619 14:32:38.036638 10109 ts_tablet_manager.cc:1397] T 8ac654811cc549d8ad2179b11413097d P 625146abfd2c4086badda901ef08c98c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 14:32:38.036875 10110 tablet_bootstrap.cc:492] T 8ac654811cc549d8ad2179b11413097d P d59bfc2b333a46bf94f4b782a7e388cd: Bootstrap starting.
I20260619 14:32:38.037006 10109 raft_consensus.cc:348] T 8ac654811cc549d8ad2179b11413097d P 625146abfd2c4086badda901ef08c98c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db3354f894ef40e3a566e882876af4f4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37821 } } peers { permanent_uuid: "625146abfd2c4086badda901ef08c98c" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 40949 } } peers { permanent_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44701 } }
I20260619 14:32:38.037109 10109 raft_consensus.cc:374] T 8ac654811cc549d8ad2179b11413097d P 625146abfd2c4086badda901ef08c98c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:38.037137 10109 raft_consensus.cc:729] T 8ac654811cc549d8ad2179b11413097d P 625146abfd2c4086badda901ef08c98c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 625146abfd2c4086badda901ef08c98c, State: Initialized, Role: FOLLOWER
I20260619 14:32:38.037263 10109 consensus_queue.cc:260] T 8ac654811cc549d8ad2179b11413097d P 625146abfd2c4086badda901ef08c98c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db3354f894ef40e3a566e882876af4f4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37821 } } peers { permanent_uuid: "625146abfd2c4086badda901ef08c98c" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 40949 } } peers { permanent_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44701 } }
I20260619 14:32:38.037510 10109 ts_tablet_manager.cc:1428] T 8ac654811cc549d8ad2179b11413097d P 625146abfd2c4086badda901ef08c98c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:32:38.037803 10108 tablet_bootstrap.cc:492] T 8ac654811cc549d8ad2179b11413097d P db3354f894ef40e3a566e882876af4f4: No bootstrap required, opened a new log
I20260619 14:32:38.037828 10110 tablet_bootstrap.cc:654] T 8ac654811cc549d8ad2179b11413097d P d59bfc2b333a46bf94f4b782a7e388cd: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:38.037880 10108 ts_tablet_manager.cc:1397] T 8ac654811cc549d8ad2179b11413097d P db3354f894ef40e3a566e882876af4f4: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 14:32:38.038443 10108 raft_consensus.cc:348] T 8ac654811cc549d8ad2179b11413097d P db3354f894ef40e3a566e882876af4f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db3354f894ef40e3a566e882876af4f4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37821 } } peers { permanent_uuid: "625146abfd2c4086badda901ef08c98c" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 40949 } } peers { permanent_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44701 } }
I20260619 14:32:38.038549 10108 raft_consensus.cc:374] T 8ac654811cc549d8ad2179b11413097d P db3354f894ef40e3a566e882876af4f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:38.038575 10108 raft_consensus.cc:729] T 8ac654811cc549d8ad2179b11413097d P db3354f894ef40e3a566e882876af4f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db3354f894ef40e3a566e882876af4f4, State: Initialized, Role: FOLLOWER
I20260619 14:32:38.038702 10108 consensus_queue.cc:260] T 8ac654811cc549d8ad2179b11413097d P db3354f894ef40e3a566e882876af4f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db3354f894ef40e3a566e882876af4f4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37821 } } peers { permanent_uuid: "625146abfd2c4086badda901ef08c98c" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 40949 } } peers { permanent_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44701 } }
I20260619 14:32:38.038992 10108 ts_tablet_manager.cc:1428] T 8ac654811cc549d8ad2179b11413097d P db3354f894ef40e3a566e882876af4f4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:32:38.039425 10110 tablet_bootstrap.cc:492] T 8ac654811cc549d8ad2179b11413097d P d59bfc2b333a46bf94f4b782a7e388cd: No bootstrap required, opened a new log
I20260619 14:32:38.039503 10110 ts_tablet_manager.cc:1397] T 8ac654811cc549d8ad2179b11413097d P d59bfc2b333a46bf94f4b782a7e388cd: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260619 14:32:38.039979 10110 raft_consensus.cc:348] T 8ac654811cc549d8ad2179b11413097d P d59bfc2b333a46bf94f4b782a7e388cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db3354f894ef40e3a566e882876af4f4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37821 } } peers { permanent_uuid: "625146abfd2c4086badda901ef08c98c" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 40949 } } peers { permanent_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44701 } }
I20260619 14:32:38.040060 10110 raft_consensus.cc:374] T 8ac654811cc549d8ad2179b11413097d P d59bfc2b333a46bf94f4b782a7e388cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:38.040081 10110 raft_consensus.cc:729] T 8ac654811cc549d8ad2179b11413097d P d59bfc2b333a46bf94f4b782a7e388cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d59bfc2b333a46bf94f4b782a7e388cd, State: Initialized, Role: FOLLOWER
I20260619 14:32:38.040198 10110 consensus_queue.cc:260] T 8ac654811cc549d8ad2179b11413097d P d59bfc2b333a46bf94f4b782a7e388cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db3354f894ef40e3a566e882876af4f4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37821 } } peers { permanent_uuid: "625146abfd2c4086badda901ef08c98c" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 40949 } } peers { permanent_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44701 } }
I20260619 14:32:38.040477 10110 ts_tablet_manager.cc:1428] T 8ac654811cc549d8ad2179b11413097d P d59bfc2b333a46bf94f4b782a7e388cd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 14:32:38.044970 10118 raft_consensus.cc:482] T 8ac654811cc549d8ad2179b11413097d P 625146abfd2c4086badda901ef08c98c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:32:38.045073 10118 raft_consensus.cc:504] T 8ac654811cc549d8ad2179b11413097d P 625146abfd2c4086badda901ef08c98c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db3354f894ef40e3a566e882876af4f4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37821 } } peers { permanent_uuid: "625146abfd2c4086badda901ef08c98c" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 40949 } } peers { permanent_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44701 } }
I20260619 14:32:38.045497 10118 leader_election.cc:288] T 8ac654811cc549d8ad2179b11413097d P 625146abfd2c4086badda901ef08c98c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db3354f894ef40e3a566e882876af4f4 (127.2.2.1:37821), d59bfc2b333a46bf94f4b782a7e388cd (127.2.2.3:44701)
I20260619 14:32:38.045853 9754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac654811cc549d8ad2179b11413097d" candidate_uuid: "625146abfd2c4086badda901ef08c98c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db3354f894ef40e3a566e882876af4f4" is_pre_election: true
I20260619 14:32:38.046000 9754 raft_consensus.cc:2454] T 8ac654811cc549d8ad2179b11413097d P db3354f894ef40e3a566e882876af4f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 625146abfd2c4086badda901ef08c98c in term 0.
I20260619 14:32:38.046315 9828 leader_election.cc:302] T 8ac654811cc549d8ad2179b11413097d P 625146abfd2c4086badda901ef08c98c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 625146abfd2c4086badda901ef08c98c, db3354f894ef40e3a566e882876af4f4; no voters:
I20260619 14:32:38.046492 10033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac654811cc549d8ad2179b11413097d" candidate_uuid: "625146abfd2c4086badda901ef08c98c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" is_pre_election: true
I20260619 14:32:38.046643 10033 raft_consensus.cc:2454] T 8ac654811cc549d8ad2179b11413097d P d59bfc2b333a46bf94f4b782a7e388cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 625146abfd2c4086badda901ef08c98c in term 0.
I20260619 14:32:38.046828 10118 raft_consensus.cc:2781] T 8ac654811cc549d8ad2179b11413097d P 625146abfd2c4086badda901ef08c98c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:32:38.046913 10118 raft_consensus.cc:482] T 8ac654811cc549d8ad2179b11413097d P 625146abfd2c4086badda901ef08c98c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:32:38.046967 10118 raft_consensus.cc:3037] T 8ac654811cc549d8ad2179b11413097d P 625146abfd2c4086badda901ef08c98c [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:38.048079 10118 raft_consensus.cc:504] T 8ac654811cc549d8ad2179b11413097d P 625146abfd2c4086badda901ef08c98c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db3354f894ef40e3a566e882876af4f4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37821 } } peers { permanent_uuid: "625146abfd2c4086badda901ef08c98c" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 40949 } } peers { permanent_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44701 } }
I20260619 14:32:38.048448 10118 leader_election.cc:288] T 8ac654811cc549d8ad2179b11413097d P 625146abfd2c4086badda901ef08c98c [CANDIDATE]: Term 1 election: Requested vote from peers db3354f894ef40e3a566e882876af4f4 (127.2.2.1:37821), d59bfc2b333a46bf94f4b782a7e388cd (127.2.2.3:44701)
I20260619 14:32:38.048784 9754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac654811cc549d8ad2179b11413097d" candidate_uuid: "625146abfd2c4086badda901ef08c98c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db3354f894ef40e3a566e882876af4f4"
I20260619 14:32:38.048851 10033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac654811cc549d8ad2179b11413097d" candidate_uuid: "625146abfd2c4086badda901ef08c98c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d59bfc2b333a46bf94f4b782a7e388cd"
I20260619 14:32:38.048903 9754 raft_consensus.cc:3037] T 8ac654811cc549d8ad2179b11413097d P db3354f894ef40e3a566e882876af4f4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:38.048960 10033 raft_consensus.cc:3037] T 8ac654811cc549d8ad2179b11413097d P d59bfc2b333a46bf94f4b782a7e388cd [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:38.049955 10033 raft_consensus.cc:2454] T 8ac654811cc549d8ad2179b11413097d P d59bfc2b333a46bf94f4b782a7e388cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 625146abfd2c4086badda901ef08c98c in term 1.
I20260619 14:32:38.050017 9754 raft_consensus.cc:2454] T 8ac654811cc549d8ad2179b11413097d P db3354f894ef40e3a566e882876af4f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 625146abfd2c4086badda901ef08c98c in term 1.
I20260619 14:32:38.050290 9828 leader_election.cc:302] T 8ac654811cc549d8ad2179b11413097d P 625146abfd2c4086badda901ef08c98c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 625146abfd2c4086badda901ef08c98c, d59bfc2b333a46bf94f4b782a7e388cd; no voters:
I20260619 14:32:38.050555 10118 raft_consensus.cc:2781] T 8ac654811cc549d8ad2179b11413097d P 625146abfd2c4086badda901ef08c98c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:32:38.050664 10118 raft_consensus.cc:686] T 8ac654811cc549d8ad2179b11413097d P 625146abfd2c4086badda901ef08c98c [term 1 LEADER]: Becoming Leader. State: Replica: 625146abfd2c4086badda901ef08c98c, State: Running, Role: LEADER
I20260619 14:32:38.050815 10118 consensus_queue.cc:237] T 8ac654811cc549d8ad2179b11413097d P 625146abfd2c4086badda901ef08c98c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db3354f894ef40e3a566e882876af4f4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37821 } } peers { permanent_uuid: "625146abfd2c4086badda901ef08c98c" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 40949 } } peers { permanent_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44701 } }
I20260619 14:32:38.052495 9596 catalog_manager.cc:5495] T 8ac654811cc549d8ad2179b11413097d P 625146abfd2c4086badda901ef08c98c reported cstate change: term changed from 0 to 1, leader changed from <none> to 625146abfd2c4086badda901ef08c98c (127.2.2.2). New cstate: current_term: 1 leader_uuid: "625146abfd2c4086badda901ef08c98c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db3354f894ef40e3a566e882876af4f4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "625146abfd2c4086badda901ef08c98c" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 40949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44701 } health_report { overall_health: UNKNOWN } } }
I20260619 14:32:38.063726 10103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:38.061006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50802 (local address 127.2.2.2:40949)
0619 14:32:38.061123 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:38.061126 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:38.061139 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:32:38.061230 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:38.061233 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:38.061284 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:38.061349 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:32:38.061559 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:38.062046 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:38.062895 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:38.063098 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:38.063174 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:38.063383 (+ 209us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:32:38.063448 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:32:38.063500 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:38.063581 (+ 81us) server_negotiation.cc:299] Negotiation successful
0619 14:32:38.063621 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 14:32:38.065613 10033 raft_consensus.cc:1261] T 8ac654811cc549d8ad2179b11413097d P d59bfc2b333a46bf94f4b782a7e388cd [term 1 FOLLOWER]: Refusing update from remote peer 625146abfd2c4086badda901ef08c98c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:32:38.065632 9754 raft_consensus.cc:1261] T 8ac654811cc549d8ad2179b11413097d P db3354f894ef40e3a566e882876af4f4 [term 1 FOLLOWER]: Refusing update from remote peer 625146abfd2c4086badda901ef08c98c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:32:38.066028 10118 consensus_queue.cc:1035] T 8ac654811cc549d8ad2179b11413097d P 625146abfd2c4086badda901ef08c98c [LEADER]: Connected to new peer: Peer: permanent_uuid: "db3354f894ef40e3a566e882876af4f4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:32:38.066254 10115 consensus_queue.cc:1035] T 8ac654811cc549d8ad2179b11413097d P 625146abfd2c4086badda901ef08c98c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:32:38.074481 9591 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33116:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:32:38.074622 9591 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33116:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:32:38.076488 9591 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 50728c18eca74389b5c824d5d291fa09: Sending DeleteTablet for 3 replicas of tablet 8ac654811cc549d8ad2179b11413097d
I20260619 14:32:38.077056 9734 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ac654811cc549d8ad2179b11413097d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:32:38 UTC) from {username='oryx', principal='oryx/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:42494
I20260619 14:32:38.077159 9874 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ac654811cc549d8ad2179b11413097d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:32:38 UTC) from {username='oryx', principal='oryx/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:50778
I20260619 14:32:38.077172 10014 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ac654811cc549d8ad2179b11413097d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:32:38 UTC) from {username='oryx', principal='oryx/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:50450
I20260619 14:32:38.077253 10133 tablet_replica.cc:330] T 8ac654811cc549d8ad2179b11413097d P db3354f894ef40e3a566e882876af4f4: stopping tablet replica
I20260619 14:32:38.077339 10134 tablet_replica.cc:330] T 8ac654811cc549d8ad2179b11413097d P 625146abfd2c4086badda901ef08c98c: stopping tablet replica
I20260619 14:32:38.077369 10132 tablet_replica.cc:330] T 8ac654811cc549d8ad2179b11413097d P d59bfc2b333a46bf94f4b782a7e388cd: stopping tablet replica
I20260619 14:32:38.077384 10133 raft_consensus.cc:2229] T 8ac654811cc549d8ad2179b11413097d P db3354f894ef40e3a566e882876af4f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:32:38.077447 10134 raft_consensus.cc:2229] T 8ac654811cc549d8ad2179b11413097d P 625146abfd2c4086badda901ef08c98c [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:32:38.077505 10132 raft_consensus.cc:2229] T 8ac654811cc549d8ad2179b11413097d P d59bfc2b333a46bf94f4b782a7e388cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:32:38.077515 10133 raft_consensus.cc:2258] T 8ac654811cc549d8ad2179b11413097d P db3354f894ef40e3a566e882876af4f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:32:38.077641 10132 raft_consensus.cc:2258] T 8ac654811cc549d8ad2179b11413097d P d59bfc2b333a46bf94f4b782a7e388cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:32:38.077656 10134 raft_consensus.cc:2258] T 8ac654811cc549d8ad2179b11413097d P 625146abfd2c4086badda901ef08c98c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:32:38.078267 10133 ts_tablet_manager.cc:1905] T 8ac654811cc549d8ad2179b11413097d P db3354f894ef40e3a566e882876af4f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:32:38.078409 10134 ts_tablet_manager.cc:1905] T 8ac654811cc549d8ad2179b11413097d P 625146abfd2c4086badda901ef08c98c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:32:38.078581 9591 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33116:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:32:38.078996 9591 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:32:38.080121 10132 ts_tablet_manager.cc:1905] T 8ac654811cc549d8ad2179b11413097d P d59bfc2b333a46bf94f4b782a7e388cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:32:38.080652 10134 ts_tablet_manager.cc:1918] T 8ac654811cc549d8ad2179b11413097d P 625146abfd2c4086badda901ef08c98c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:32:38.080745 10134 log.cc:1196] T 8ac654811cc549d8ad2179b11413097d P 625146abfd2c4086badda901ef08c98c: Deleting WAL directory at /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/ts-1/wal/wals/8ac654811cc549d8ad2179b11413097d
I20260619 14:32:38.081014 10134 ts_tablet_manager.cc:1939] T 8ac654811cc549d8ad2179b11413097d P 625146abfd2c4086badda901ef08c98c: Deleting consensus metadata
I20260619 14:32:38.081380 10133 ts_tablet_manager.cc:1918] T 8ac654811cc549d8ad2179b11413097d P db3354f894ef40e3a566e882876af4f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:32:38.081506 10133 log.cc:1196] T 8ac654811cc549d8ad2179b11413097d P db3354f894ef40e3a566e882876af4f4: Deleting WAL directory at /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/ts-0/wal/wals/8ac654811cc549d8ad2179b11413097d
I20260619 14:32:38.081766 10133 ts_tablet_manager.cc:1939] T 8ac654811cc549d8ad2179b11413097d P db3354f894ef40e3a566e882876af4f4: Deleting consensus metadata
I20260619 14:32:38.081967 9582 catalog_manager.cc:4841] TS 625146abfd2c4086badda901ef08c98c (127.2.2.2:40949): tablet 8ac654811cc549d8ad2179b11413097d (table test-table [id=953c94a3ddf7490d8cb3e3b201f80b10]) successfully deleted
I20260619 14:32:38.082767 10132 ts_tablet_manager.cc:1918] T 8ac654811cc549d8ad2179b11413097d P d59bfc2b333a46bf94f4b782a7e388cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:32:38.082854 10132 log.cc:1196] T 8ac654811cc549d8ad2179b11413097d P d59bfc2b333a46bf94f4b782a7e388cd: Deleting WAL directory at /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/ts-2/wal/wals/8ac654811cc549d8ad2179b11413097d
I20260619 14:32:38.082855 9584 catalog_manager.cc:4841] TS db3354f894ef40e3a566e882876af4f4 (127.2.2.1:37821): tablet 8ac654811cc549d8ad2179b11413097d (table test-table [id=953c94a3ddf7490d8cb3e3b201f80b10]) successfully deleted
I20260619 14:32:38.083108 10132 ts_tablet_manager.cc:1939] T 8ac654811cc549d8ad2179b11413097d P d59bfc2b333a46bf94f4b782a7e388cd: Deleting consensus metadata
I20260619 14:32:38.083820 9584 catalog_manager.cc:4841] TS d59bfc2b333a46bf94f4b782a7e388cd (127.2.2.3:44701): tablet 8ac654811cc549d8ad2179b11413097d (table test-table [id=953c94a3ddf7490d8cb3e3b201f80b10]) successfully deleted
I20260619 14:32:38.084904 9734 tablet_service.cc:1467] Processing CreateTablet for tablet 0c87ddda443f474caccbbb6cd33f2c0a (DEFAULT_TABLE table=test-table [id=ec7f8009766945eb946c5e2e6a01c0a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:32:38.084919 9874 tablet_service.cc:1467] Processing CreateTablet for tablet 0c87ddda443f474caccbbb6cd33f2c0a (DEFAULT_TABLE table=test-table [id=ec7f8009766945eb946c5e2e6a01c0a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:32:38.084995 10014 tablet_service.cc:1467] Processing CreateTablet for tablet 0c87ddda443f474caccbbb6cd33f2c0a (DEFAULT_TABLE table=test-table [id=ec7f8009766945eb946c5e2e6a01c0a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:32:38.085171 9734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c87ddda443f474caccbbb6cd33f2c0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:38.085211 10014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c87ddda443f474caccbbb6cd33f2c0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:38.085253 9874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c87ddda443f474caccbbb6cd33f2c0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:38.087180 10109 tablet_bootstrap.cc:492] T 0c87ddda443f474caccbbb6cd33f2c0a P 625146abfd2c4086badda901ef08c98c: Bootstrap starting.
I20260619 14:32:38.087383 10110 tablet_bootstrap.cc:492] T 0c87ddda443f474caccbbb6cd33f2c0a P d59bfc2b333a46bf94f4b782a7e388cd: Bootstrap starting.
I20260619 14:32:38.087383 10108 tablet_bootstrap.cc:492] T 0c87ddda443f474caccbbb6cd33f2c0a P db3354f894ef40e3a566e882876af4f4: Bootstrap starting.
I20260619 14:32:38.087987 10109 tablet_bootstrap.cc:654] T 0c87ddda443f474caccbbb6cd33f2c0a P 625146abfd2c4086badda901ef08c98c: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:38.088186 10108 tablet_bootstrap.cc:654] T 0c87ddda443f474caccbbb6cd33f2c0a P db3354f894ef40e3a566e882876af4f4: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:38.088207 10110 tablet_bootstrap.cc:654] T 0c87ddda443f474caccbbb6cd33f2c0a P d59bfc2b333a46bf94f4b782a7e388cd: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:38.089303 10110 tablet_bootstrap.cc:492] T 0c87ddda443f474caccbbb6cd33f2c0a P d59bfc2b333a46bf94f4b782a7e388cd: No bootstrap required, opened a new log
I20260619 14:32:38.089355 10108 tablet_bootstrap.cc:492] T 0c87ddda443f474caccbbb6cd33f2c0a P db3354f894ef40e3a566e882876af4f4: No bootstrap required, opened a new log
I20260619 14:32:38.089390 10110 ts_tablet_manager.cc:1397] T 0c87ddda443f474caccbbb6cd33f2c0a P d59bfc2b333a46bf94f4b782a7e388cd: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 14:32:38.089417 10108 ts_tablet_manager.cc:1397] T 0c87ddda443f474caccbbb6cd33f2c0a P db3354f894ef40e3a566e882876af4f4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 14:32:38.089807 10108 raft_consensus.cc:348] T 0c87ddda443f474caccbbb6cd33f2c0a P db3354f894ef40e3a566e882876af4f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db3354f894ef40e3a566e882876af4f4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37821 } } peers { permanent_uuid: "625146abfd2c4086badda901ef08c98c" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 40949 } } peers { permanent_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44701 } }
I20260619 14:32:38.089810 10110 raft_consensus.cc:348] T 0c87ddda443f474caccbbb6cd33f2c0a P d59bfc2b333a46bf94f4b782a7e388cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db3354f894ef40e3a566e882876af4f4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37821 } } peers { permanent_uuid: "625146abfd2c4086badda901ef08c98c" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 40949 } } peers { permanent_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44701 } }
I20260619 14:32:38.089924 10108 raft_consensus.cc:374] T 0c87ddda443f474caccbbb6cd33f2c0a P db3354f894ef40e3a566e882876af4f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:38.089932 10110 raft_consensus.cc:374] T 0c87ddda443f474caccbbb6cd33f2c0a P d59bfc2b333a46bf94f4b782a7e388cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:38.089989 10108 raft_consensus.cc:729] T 0c87ddda443f474caccbbb6cd33f2c0a P db3354f894ef40e3a566e882876af4f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db3354f894ef40e3a566e882876af4f4, State: Initialized, Role: FOLLOWER
I20260619 14:32:38.089989 10110 raft_consensus.cc:729] T 0c87ddda443f474caccbbb6cd33f2c0a P d59bfc2b333a46bf94f4b782a7e388cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d59bfc2b333a46bf94f4b782a7e388cd, State: Initialized, Role: FOLLOWER
I20260619 14:32:38.090137 10110 consensus_queue.cc:260] T 0c87ddda443f474caccbbb6cd33f2c0a P d59bfc2b333a46bf94f4b782a7e388cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db3354f894ef40e3a566e882876af4f4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37821 } } peers { permanent_uuid: "625146abfd2c4086badda901ef08c98c" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 40949 } } peers { permanent_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44701 } }
I20260619 14:32:38.090149 10108 consensus_queue.cc:260] T 0c87ddda443f474caccbbb6cd33f2c0a P db3354f894ef40e3a566e882876af4f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db3354f894ef40e3a566e882876af4f4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37821 } } peers { permanent_uuid: "625146abfd2c4086badda901ef08c98c" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 40949 } } peers { permanent_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44701 } }
I20260619 14:32:38.090418 10110 ts_tablet_manager.cc:1428] T 0c87ddda443f474caccbbb6cd33f2c0a P d59bfc2b333a46bf94f4b782a7e388cd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:32:38.090420 10108 ts_tablet_manager.cc:1428] T 0c87ddda443f474caccbbb6cd33f2c0a P db3354f894ef40e3a566e882876af4f4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:32:38.091974 10109 tablet_bootstrap.cc:492] T 0c87ddda443f474caccbbb6cd33f2c0a P 625146abfd2c4086badda901ef08c98c: No bootstrap required, opened a new log
I20260619 14:32:38.092068 10109 ts_tablet_manager.cc:1397] T 0c87ddda443f474caccbbb6cd33f2c0a P 625146abfd2c4086badda901ef08c98c: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260619 14:32:38.092501 10109 raft_consensus.cc:348] T 0c87ddda443f474caccbbb6cd33f2c0a P 625146abfd2c4086badda901ef08c98c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db3354f894ef40e3a566e882876af4f4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37821 } } peers { permanent_uuid: "625146abfd2c4086badda901ef08c98c" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 40949 } } peers { permanent_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44701 } }
I20260619 14:32:38.092614 10109 raft_consensus.cc:374] T 0c87ddda443f474caccbbb6cd33f2c0a P 625146abfd2c4086badda901ef08c98c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:38.092646 10109 raft_consensus.cc:729] T 0c87ddda443f474caccbbb6cd33f2c0a P 625146abfd2c4086badda901ef08c98c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 625146abfd2c4086badda901ef08c98c, State: Initialized, Role: FOLLOWER
I20260619 14:32:38.092772 10109 consensus_queue.cc:260] T 0c87ddda443f474caccbbb6cd33f2c0a P 625146abfd2c4086badda901ef08c98c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db3354f894ef40e3a566e882876af4f4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37821 } } peers { permanent_uuid: "625146abfd2c4086badda901ef08c98c" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 40949 } } peers { permanent_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44701 } }
I20260619 14:32:38.093061 10109 ts_tablet_manager.cc:1428] T 0c87ddda443f474caccbbb6cd33f2c0a P 625146abfd2c4086badda901ef08c98c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:32:38.126499 10150 raft_consensus.cc:482] T 0c87ddda443f474caccbbb6cd33f2c0a P db3354f894ef40e3a566e882876af4f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:32:38.126641 10150 raft_consensus.cc:504] T 0c87ddda443f474caccbbb6cd33f2c0a P db3354f894ef40e3a566e882876af4f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db3354f894ef40e3a566e882876af4f4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37821 } } peers { permanent_uuid: "625146abfd2c4086badda901ef08c98c" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 40949 } } peers { permanent_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44701 } }
I20260619 14:32:38.127112 10150 leader_election.cc:288] T 0c87ddda443f474caccbbb6cd33f2c0a P db3354f894ef40e3a566e882876af4f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 625146abfd2c4086badda901ef08c98c (127.2.2.2:40949), d59bfc2b333a46bf94f4b782a7e388cd (127.2.2.3:44701)
I20260619 14:32:38.127655 10033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c87ddda443f474caccbbb6cd33f2c0a" candidate_uuid: "db3354f894ef40e3a566e882876af4f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" is_pre_election: true
I20260619 14:32:38.127642 9894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c87ddda443f474caccbbb6cd33f2c0a" candidate_uuid: "db3354f894ef40e3a566e882876af4f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "625146abfd2c4086badda901ef08c98c" is_pre_election: true
I20260619 14:32:38.127800 9894 raft_consensus.cc:2454] T 0c87ddda443f474caccbbb6cd33f2c0a P 625146abfd2c4086badda901ef08c98c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db3354f894ef40e3a566e882876af4f4 in term 0.
I20260619 14:32:38.127800 10033 raft_consensus.cc:2454] T 0c87ddda443f474caccbbb6cd33f2c0a P d59bfc2b333a46bf94f4b782a7e388cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db3354f894ef40e3a566e882876af4f4 in term 0.
I20260619 14:32:38.128140 9688 leader_election.cc:302] T 0c87ddda443f474caccbbb6cd33f2c0a P db3354f894ef40e3a566e882876af4f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d59bfc2b333a46bf94f4b782a7e388cd, db3354f894ef40e3a566e882876af4f4; no voters:
I20260619 14:32:38.128335 10150 raft_consensus.cc:2781] T 0c87ddda443f474caccbbb6cd33f2c0a P db3354f894ef40e3a566e882876af4f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:32:38.128401 10150 raft_consensus.cc:482] T 0c87ddda443f474caccbbb6cd33f2c0a P db3354f894ef40e3a566e882876af4f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:32:38.128429 10150 raft_consensus.cc:3037] T 0c87ddda443f474caccbbb6cd33f2c0a P db3354f894ef40e3a566e882876af4f4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:38.129477 10150 raft_consensus.cc:504] T 0c87ddda443f474caccbbb6cd33f2c0a P db3354f894ef40e3a566e882876af4f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db3354f894ef40e3a566e882876af4f4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37821 } } peers { permanent_uuid: "625146abfd2c4086badda901ef08c98c" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 40949 } } peers { permanent_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44701 } }
I20260619 14:32:38.129840 10150 leader_election.cc:288] T 0c87ddda443f474caccbbb6cd33f2c0a P db3354f894ef40e3a566e882876af4f4 [CANDIDATE]: Term 1 election: Requested vote from peers 625146abfd2c4086badda901ef08c98c (127.2.2.2:40949), d59bfc2b333a46bf94f4b782a7e388cd (127.2.2.3:44701)
I20260619 14:32:38.130236 9894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c87ddda443f474caccbbb6cd33f2c0a" candidate_uuid: "db3354f894ef40e3a566e882876af4f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "625146abfd2c4086badda901ef08c98c"
I20260619 14:32:38.130278 10033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c87ddda443f474caccbbb6cd33f2c0a" candidate_uuid: "db3354f894ef40e3a566e882876af4f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d59bfc2b333a46bf94f4b782a7e388cd"
I20260619 14:32:38.130352 9894 raft_consensus.cc:3037] T 0c87ddda443f474caccbbb6cd33f2c0a P 625146abfd2c4086badda901ef08c98c [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:38.130445 10033 raft_consensus.cc:3037] T 0c87ddda443f474caccbbb6cd33f2c0a P d59bfc2b333a46bf94f4b782a7e388cd [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:38.131352 9894 raft_consensus.cc:2454] T 0c87ddda443f474caccbbb6cd33f2c0a P 625146abfd2c4086badda901ef08c98c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db3354f894ef40e3a566e882876af4f4 in term 1.
I20260619 14:32:38.131461 10033 raft_consensus.cc:2454] T 0c87ddda443f474caccbbb6cd33f2c0a P d59bfc2b333a46bf94f4b782a7e388cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db3354f894ef40e3a566e882876af4f4 in term 1.
I20260619 14:32:38.131721 9686 leader_election.cc:302] T 0c87ddda443f474caccbbb6cd33f2c0a P db3354f894ef40e3a566e882876af4f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 625146abfd2c4086badda901ef08c98c, db3354f894ef40e3a566e882876af4f4; no voters:
I20260619 14:32:38.131955 10150 raft_consensus.cc:2781] T 0c87ddda443f474caccbbb6cd33f2c0a P db3354f894ef40e3a566e882876af4f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:32:38.132067 10150 raft_consensus.cc:686] T 0c87ddda443f474caccbbb6cd33f2c0a P db3354f894ef40e3a566e882876af4f4 [term 1 LEADER]: Becoming Leader. State: Replica: db3354f894ef40e3a566e882876af4f4, State: Running, Role: LEADER
I20260619 14:32:38.132215 10150 consensus_queue.cc:237] T 0c87ddda443f474caccbbb6cd33f2c0a P db3354f894ef40e3a566e882876af4f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db3354f894ef40e3a566e882876af4f4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37821 } } peers { permanent_uuid: "625146abfd2c4086badda901ef08c98c" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 40949 } } peers { permanent_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44701 } }
I20260619 14:32:38.133946 9596 catalog_manager.cc:5495] T 0c87ddda443f474caccbbb6cd33f2c0a P db3354f894ef40e3a566e882876af4f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to db3354f894ef40e3a566e882876af4f4 (127.2.2.1). New cstate: current_term: 1 leader_uuid: "db3354f894ef40e3a566e882876af4f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db3354f894ef40e3a566e882876af4f4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 37821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "625146abfd2c4086badda901ef08c98c" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 40949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44701 } health_report { overall_health: UNKNOWN } } }
I20260619 14:32:38.151279 10107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:38.147800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42520 (local address 127.2.2.1:37821)
0619 14:32:38.147914 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:38.147918 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:38.147933 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:32:38.148029 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:38.148032 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:38.148079 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:38.148161 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:32:38.148351 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:38.149146 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:38.150159 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:38.150505 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:38.150612 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:38.150865 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:32:38.150948 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:32:38.151017 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:38.151090 (+ 73us) server_negotiation.cc:299] Negotiation successful
0619 14:32:38.151144 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 14:32:38.160212 10107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:38.157065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:40225 (local address 127.2.2.1:37821)
0619 14:32:38.157169 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:38.157173 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:38.157184 (+ 11us) server_negotiation.cc:407] Connection header received
0619 14:32:38.157308 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:38.157310 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:38.157348 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:38.157406 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:32:38.157617 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:38.158087 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:38.159263 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:38.159463 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:38.159536 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:38.159780 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:32:38.159841 (+ 61us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:32:38.159892 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:38.160002 (+ 110us) server_negotiation.cc:299] Negotiation successful
0619 14:32:38.160073 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":40}
I20260619 14:32:38.160243 10164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:38.156984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.1:37821 (local address 127.2.2.1:40225)
0619 14:32:38.157129 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:38.157143 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:38.157218 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:38.157430 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:38.157433 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:38.157439 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:32:38.157529 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:38.157533 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:38.158234 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:38.158237 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:38.159151 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:38.159160 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:38.159548 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:38.159564 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:38.159646 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:38.159675 (+ 29us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:32:38.159915 (+ 240us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:32:38.159921 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 14:32:38.159983 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 14:32:38.160084 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":1}
I20260619 14:32:38.166355 10033 raft_consensus.cc:1261] T 0c87ddda443f474caccbbb6cd33f2c0a P d59bfc2b333a46bf94f4b782a7e388cd [term 1 FOLLOWER]: Refusing update from remote peer db3354f894ef40e3a566e882876af4f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:32:38.166374 9894 raft_consensus.cc:1261] T 0c87ddda443f474caccbbb6cd33f2c0a P 625146abfd2c4086badda901ef08c98c [term 1 FOLLOWER]: Refusing update from remote peer db3354f894ef40e3a566e882876af4f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:32:38.166788 10150 consensus_queue.cc:1035] T 0c87ddda443f474caccbbb6cd33f2c0a P db3354f894ef40e3a566e882876af4f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d59bfc2b333a46bf94f4b782a7e388cd" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 44701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:32:38.166972 10162 consensus_queue.cc:1035] T 0c87ddda443f474caccbbb6cd33f2c0a P db3354f894ef40e3a566e882876af4f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "625146abfd2c4086badda901ef08c98c" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 40949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:32:38.198482 9596 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33116:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:32:38.198638 9596 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33116:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:32:38.200762 9596 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 50728c18eca74389b5c824d5d291fa09: Sending DeleteTablet for 3 replicas of tablet 0c87ddda443f474caccbbb6cd33f2c0a
I20260619 14:32:38.201419 10014 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c87ddda443f474caccbbb6cd33f2c0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:32:38 UTC) from {username='oryx', principal='oryx/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:50450
I20260619 14:32:38.201402 9874 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c87ddda443f474caccbbb6cd33f2c0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:32:38 UTC) from {username='oryx', principal='oryx/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:50778
I20260619 14:32:38.201638 10132 tablet_replica.cc:330] T 0c87ddda443f474caccbbb6cd33f2c0a P d59bfc2b333a46bf94f4b782a7e388cd: stopping tablet replica
I20260619 14:32:38.201736 9734 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c87ddda443f474caccbbb6cd33f2c0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:32:38 UTC) from {username='oryx', principal='oryx/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:42494
I20260619 14:32:38.201769 10132 raft_consensus.cc:2229] T 0c87ddda443f474caccbbb6cd33f2c0a P d59bfc2b333a46bf94f4b782a7e388cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:32:38.201884 10132 raft_consensus.cc:2258] T 0c87ddda443f474caccbbb6cd33f2c0a P d59bfc2b333a46bf94f4b782a7e388cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:32:38.201929 10133 tablet_replica.cc:330] T 0c87ddda443f474caccbbb6cd33f2c0a P db3354f894ef40e3a566e882876af4f4: stopping tablet replica
I20260619 14:32:38.202040 10133 raft_consensus.cc:2229] T 0c87ddda443f474caccbbb6cd33f2c0a P db3354f894ef40e3a566e882876af4f4 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:32:38.202061 10134 tablet_replica.cc:330] T 0c87ddda443f474caccbbb6cd33f2c0a P 625146abfd2c4086badda901ef08c98c: stopping tablet replica
I20260619 14:32:38.202193 10134 raft_consensus.cc:2229] T 0c87ddda443f474caccbbb6cd33f2c0a P 625146abfd2c4086badda901ef08c98c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:32:38.202317 10134 raft_consensus.cc:2258] T 0c87ddda443f474caccbbb6cd33f2c0a P 625146abfd2c4086badda901ef08c98c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:32:38.202421 10133 raft_consensus.cc:2258] T 0c87ddda443f474caccbbb6cd33f2c0a P db3354f894ef40e3a566e882876af4f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:32:38.202889 10132 ts_tablet_manager.cc:1905] T 0c87ddda443f474caccbbb6cd33f2c0a P d59bfc2b333a46bf94f4b782a7e388cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:32:38.203083 10133 ts_tablet_manager.cc:1905] T 0c87ddda443f474caccbbb6cd33f2c0a P db3354f894ef40e3a566e882876af4f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:32:38.205518 10132 ts_tablet_manager.cc:1918] T 0c87ddda443f474caccbbb6cd33f2c0a P d59bfc2b333a46bf94f4b782a7e388cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:32:38.205518 10133 ts_tablet_manager.cc:1918] T 0c87ddda443f474caccbbb6cd33f2c0a P db3354f894ef40e3a566e882876af4f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:32:38.205598 10132 log.cc:1196] T 0c87ddda443f474caccbbb6cd33f2c0a P d59bfc2b333a46bf94f4b782a7e388cd: Deleting WAL directory at /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/ts-2/wal/wals/0c87ddda443f474caccbbb6cd33f2c0a
I20260619 14:32:38.205598 10133 log.cc:1196] T 0c87ddda443f474caccbbb6cd33f2c0a P db3354f894ef40e3a566e882876af4f4: Deleting WAL directory at /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/ts-0/wal/wals/0c87ddda443f474caccbbb6cd33f2c0a
I20260619 14:32:38.205862 10132 ts_tablet_manager.cc:1939] T 0c87ddda443f474caccbbb6cd33f2c0a P d59bfc2b333a46bf94f4b782a7e388cd: Deleting consensus metadata
I20260619 14:32:38.205862 10133 ts_tablet_manager.cc:1939] T 0c87ddda443f474caccbbb6cd33f2c0a P db3354f894ef40e3a566e882876af4f4: Deleting consensus metadata
I20260619 14:32:38.206791 9584 catalog_manager.cc:4841] TS d59bfc2b333a46bf94f4b782a7e388cd (127.2.2.3:44701): tablet 0c87ddda443f474caccbbb6cd33f2c0a (table test-table [id=ec7f8009766945eb946c5e2e6a01c0a0]) successfully deleted
I20260619 14:32:38.206925 10134 ts_tablet_manager.cc:1905] T 0c87ddda443f474caccbbb6cd33f2c0a P 625146abfd2c4086badda901ef08c98c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:32:38.207353 9584 catalog_manager.cc:4841] TS db3354f894ef40e3a566e882876af4f4 (127.2.2.1:37821): tablet 0c87ddda443f474caccbbb6cd33f2c0a (table test-table [id=ec7f8009766945eb946c5e2e6a01c0a0]) successfully deleted
I20260619 14:32:38.207795 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 9669
I20260619 14:32:38.209530 10134 ts_tablet_manager.cc:1918] T 0c87ddda443f474caccbbb6cd33f2c0a P 625146abfd2c4086badda901ef08c98c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:32:38.209640 10134 log.cc:1196] T 0c87ddda443f474caccbbb6cd33f2c0a P 625146abfd2c4086badda901ef08c98c: Deleting WAL directory at /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879514993456-2056-0/minicluster-data/ts-1/wal/wals/0c87ddda443f474caccbbb6cd33f2c0a
I20260619 14:32:38.209887 10134 ts_tablet_manager.cc:1939] T 0c87ddda443f474caccbbb6cd33f2c0a P 625146abfd2c4086badda901ef08c98c: Deleting consensus metadata
I20260619 14:32:38.210598 9582 catalog_manager.cc:4841] TS 625146abfd2c4086badda901ef08c98c (127.2.2.2:40949): tablet 0c87ddda443f474caccbbb6cd33f2c0a (table test-table [id=ec7f8009766945eb946c5e2e6a01c0a0]) successfully deleted
I20260619 14:32:38.215832 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 9809
I20260619 14:32:38.222671 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 9949
I20260619 14:32:38.228300 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 9565
2026-06-19T14:32:38Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2960 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781879514993456-2056-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:32:38 dist-test-slave-70rm krb5kdc[10196](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:32:38 dist-test-slave-70rm krb5kdc[10196](info): set up 2 sockets
Jun 19 14:32:38 dist-test-slave-70rm krb5kdc[10196](info): commencing operation
krb5kdc: starting...
W20260619 14:32:39.275417 2056 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Jun 19 14:32:39 dist-test-slave-70rm krb5kdc[10196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:32:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:32:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.2.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab.
Entry for principal kudu/127.2.2.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab.
I20260619 14:32:39.401237 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781879514993456-2056-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781879514993456-2056-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781879514993456-2056-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:36643
--webserver_interface=127.2.2.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:34007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.2.62:36643
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:32:39.511320 10209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:39.511620 10209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:39.511663 10209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:39.515993 10209 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:32:39.516064 10209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:39.516084 10209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:39.516101 10209 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:32:39.516119 10209 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:32:39.521625 10209 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:34007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781879514993456-2056-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781879514993456-2056-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.2.62:36643
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:36643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.2.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781879514993456-2056-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:32:39.522775 10209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:39.523738 10209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:32:39.532584 10214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:39.532733 10209 server_base.cc:1034] running on GCE node
W20260619 14:32:39.532584 10217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:39.532584 10215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:39.533605 10209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:39.534883 10209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:39.536103 10209 hybrid_clock.cc:648] HybridClock initialized: now 1781879559536056 us; error 35 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-19T14:32:39Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1320 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:32:39 dist-test-slave-70rm krb5kdc[10222](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:32:39 dist-test-slave-70rm krb5kdc[10222](info): set up 2 sockets
Jun 19 14:32:39 dist-test-slave-70rm krb5kdc[10222](info): commencing operation
krb5kdc: starting...
W20260619 14:32:40.622947 2056 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Jun 19 14:32:40 dist-test-slave-70rm krb5kdc[10222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:32:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:32:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.2.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab.
Entry for principal kudu/127.2.2.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab.
I20260619 14:32:40.746523 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:46287
--webserver_interface=127.2.2.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:44141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.2.62:46287
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:32:40.856796 10235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:40.857138 10235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:40.857214 10235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:40.861302 10235 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:32:40.861413 10235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:40.861451 10235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:40.861507 10235 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:32:40.861565 10235 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:32:40.867003 10235 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:44141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.2.62:46287
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:46287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.2.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:32:40.868170 10235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:40.869236 10235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:32:40.878233 10240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:40.878249 10241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:40.878350 10235 server_base.cc:1034] running on GCE node
W20260619 14:32:40.878232 10243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:40.879145 10235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:40.880327 10235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:40.881520 10235 hybrid_clock.cc:648] HybridClock initialized: now 1781879560881496 us; error 38 us; skew 500 ppm
Jun 19 14:32:40 dist-test-slave-70rm krb5kdc[10222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879560, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:32:40.888855 10235 init.cc:377] Logged in from keytab as kudu/127.2.2.62@KRBTEST.COM (short username kudu)
I20260619 14:32:40.889971 10235 webserver.cc:460] Webserver started at http://127.2.2.62:43833/ using document root <none> and password file <none>
I20260619 14:32:40.890612 10235 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:40.890687 10235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:40.890937 10235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:32:40.893151 10235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/master-0/data/instance:
uuid: "68cfe4990e2240f8a1936a97a7c6139c"
format_stamp: "Formatted at 2026-06-19 14:32:40 on dist-test-slave-70rm"
server_key: "b59f7718a0c8bd35c38d729f7585b661"
server_key_iv: "64665485422962e9448454b7361c676a"
server_key_version: "encryptionkey@0"
I20260619 14:32:40.893847 10235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/master-0/wal/instance:
uuid: "68cfe4990e2240f8a1936a97a7c6139c"
format_stamp: "Formatted at 2026-06-19 14:32:40 on dist-test-slave-70rm"
server_key: "b59f7718a0c8bd35c38d729f7585b661"
server_key_iv: "64665485422962e9448454b7361c676a"
server_key_version: "encryptionkey@0"
I20260619 14:32:40.898054 10235 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 14:32:40.900794 10250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:40.902066 10235 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 14:32:40.902240 10235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/master-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/master-0/wal
uuid: "68cfe4990e2240f8a1936a97a7c6139c"
format_stamp: "Formatted at 2026-06-19 14:32:40 on dist-test-slave-70rm"
server_key: "b59f7718a0c8bd35c38d729f7585b661"
server_key_iv: "64665485422962e9448454b7361c676a"
server_key_version: "encryptionkey@0"
I20260619 14:32:40.902364 10235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:40.915165 10235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:40.918708 10235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:32:40.918917 10235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:40.925967 10235 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.62:46287
I20260619 14:32:40.925990 10302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.62:46287 every 8 connection(s)
I20260619 14:32:40.926978 10235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
I20260619 14:32:40.929708 10303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:40.932886 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 10235
I20260619 14:32:40.932994 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/master-0/wal/instance
I20260619 14:32:40.933290 2056 external_mini_cluster.cc:1428] Setting key 9fb55d328ae2971fe9a758b55faf9c4b
I20260619 14:32:40.935855 10303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68cfe4990e2240f8a1936a97a7c6139c: Bootstrap starting.
I20260619 14:32:40.938570 10303 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68cfe4990e2240f8a1936a97a7c6139c: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:40.939431 10303 log.cc:824] T 00000000000000000000000000000000 P 68cfe4990e2240f8a1936a97a7c6139c: Log is configured to *not* fsync() on all Append() calls
I20260619 14:32:40.941708 10303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68cfe4990e2240f8a1936a97a7c6139c: No bootstrap required, opened a new log
Jun 19 14:32:40 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:32:40.946269 10303 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68cfe4990e2240f8a1936a97a7c6139c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68cfe4990e2240f8a1936a97a7c6139c" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 46287 } }
I20260619 14:32:40.946614 10303 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68cfe4990e2240f8a1936a97a7c6139c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:40.946676 10303 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68cfe4990e2240f8a1936a97a7c6139c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68cfe4990e2240f8a1936a97a7c6139c, State: Initialized, Role: FOLLOWER
I20260619 14:32:40.947175 10303 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68cfe4990e2240f8a1936a97a7c6139c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68cfe4990e2240f8a1936a97a7c6139c" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 46287 } }
I20260619 14:32:40.947304 10303 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68cfe4990e2240f8a1936a97a7c6139c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:32:40.947383 10303 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68cfe4990e2240f8a1936a97a7c6139c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:32:40.947477 10303 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68cfe4990e2240f8a1936a97a7c6139c [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:40.948400 10303 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68cfe4990e2240f8a1936a97a7c6139c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68cfe4990e2240f8a1936a97a7c6139c" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 46287 } }
I20260619 14:32:40.948766 10303 leader_election.cc:302] T 00000000000000000000000000000000 P 68cfe4990e2240f8a1936a97a7c6139c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68cfe4990e2240f8a1936a97a7c6139c; no voters:
I20260619 14:32:40.949146 10303 leader_election.cc:288] T 00000000000000000000000000000000 P 68cfe4990e2240f8a1936a97a7c6139c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:32:40.949237 10308 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68cfe4990e2240f8a1936a97a7c6139c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:32:40.949486 10308 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68cfe4990e2240f8a1936a97a7c6139c [term 1 LEADER]: Becoming Leader. State: Replica: 68cfe4990e2240f8a1936a97a7c6139c, State: Running, Role: LEADER
I20260619 14:32:40.949854 10308 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68cfe4990e2240f8a1936a97a7c6139c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68cfe4990e2240f8a1936a97a7c6139c" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 46287 } }
I20260619 14:32:40.950362 10306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:40.934777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57994 (local address 127.2.2.62:46287)
0619 14:32:40.935211 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:40.935229 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:40.935272 (+ 43us) server_negotiation.cc:407] Connection header received
0619 14:32:40.936098 (+ 826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:40.936121 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:40.936535 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:40.936910 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:40.937578 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:40.939112 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:40.940145 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:40.940836 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:40.940990 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:40.943505 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:40.943511 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:40.943528 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:40.943559 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:40.945798 (+ 2239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:40.947435 (+ 1637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:40.947441 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:40.947448 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:40.947620 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:40.947927 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:40.947931 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:40.947933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:40.948361 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:40.948582 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:40.948881 (+ 299us) server_negotiation.cc:299] Negotiation successful
0619 14:32:40.949258 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":118,"threads_started":1}
I20260619 14:32:40.950660 10303 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68cfe4990e2240f8a1936a97a7c6139c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:32:40.952431 10310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68cfe4990e2240f8a1936a97a7c6139c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68cfe4990e2240f8a1936a97a7c6139c. Latest consensus state: current_term: 1 leader_uuid: "68cfe4990e2240f8a1936a97a7c6139c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68cfe4990e2240f8a1936a97a7c6139c" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 46287 } } }
I20260619 14:32:40.952549 10310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68cfe4990e2240f8a1936a97a7c6139c [sys.catalog]: This master's current role is: LEADER
I20260619 14:32:40.952860 10309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68cfe4990e2240f8a1936a97a7c6139c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68cfe4990e2240f8a1936a97a7c6139c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68cfe4990e2240f8a1936a97a7c6139c" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 46287 } } }
I20260619 14:32:40.953032 10309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68cfe4990e2240f8a1936a97a7c6139c [sys.catalog]: This master's current role is: LEADER
I20260619 14:32:40.953356 10323 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:32:40.956069 10323 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:32:40.961627 10323 catalog_manager.cc:1347] Generated new cluster ID: e43c144fac7e4be292d6ddc814ae0217
I20260619 14:32:40.961716 10323 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:32:40.976639 10323 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:32:40.977985 10323 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:32:40.990994 10323 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68cfe4990e2240f8a1936a97a7c6139c: Generated new TSK 0
I20260619 14:32:40.991909 10323 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.2.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab.
Entry for principal kudu/127.2.2.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab.
I20260619 14:32:41.021207 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.1:0
--local_ip_for_outbound_sockets=127.2.2.1
--webserver_interface=127.2.2.1
--webserver_port=0
--tserver_master_addrs=127.2.2.62:46287
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab
--principal=kudu/127.2.2.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:44141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:32:41.134585 10329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:41.134917 10329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:41.134992 10329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:41.139241 10329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:41.139331 10329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:41.139420 10329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.1
I20260619 14:32:41.144821 10329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:44141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab
--principal=kudu/127.2.2.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.2.1
--webserver_port=0
--tserver_master_addrs=127.2.2.62:46287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.1
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:32:41.145958 10329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:41.147352 10329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:32:41.157595 10337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:41.157590 10335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:41.157806 10329 server_base.cc:1034] running on GCE node
W20260619 14:32:41.157968 10334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:41.158499 10329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:41.159226 10329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:41.160432 10329 hybrid_clock.cc:648] HybridClock initialized: now 1781879561160412 us; error 34 us; skew 500 ppm
Jun 19 14:32:41 dist-test-slave-70rm krb5kdc[10222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879561, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:32:41.166427 10329 init.cc:377] Logged in from keytab as kudu/127.2.2.1@KRBTEST.COM (short username kudu)
I20260619 14:32:41.167505 10329 webserver.cc:460] Webserver started at http://127.2.2.1:35585/ using document root <none> and password file <none>
I20260619 14:32:41.168138 10329 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:41.168242 10329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:41.168496 10329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:32:41.170852 10329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-0/data/instance:
uuid: "5cb491f5b66d465e821b3f1470302c41"
format_stamp: "Formatted at 2026-06-19 14:32:41 on dist-test-slave-70rm"
server_key: "d1f7c1df53dcd1a1e5ac1ee854aaa502"
server_key_iv: "14d3df81b67b3088e6c725f1c9ea0385"
server_key_version: "encryptionkey@0"
I20260619 14:32:41.171500 10329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-0/wal/instance:
uuid: "5cb491f5b66d465e821b3f1470302c41"
format_stamp: "Formatted at 2026-06-19 14:32:41 on dist-test-slave-70rm"
server_key: "d1f7c1df53dcd1a1e5ac1ee854aaa502"
server_key_iv: "14d3df81b67b3088e6c725f1c9ea0385"
server_key_version: "encryptionkey@0"
I20260619 14:32:41.175609 10329 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:32:41.178376 10344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:41.179719 10329 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:32:41.179957 10329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-0/wal
uuid: "5cb491f5b66d465e821b3f1470302c41"
format_stamp: "Formatted at 2026-06-19 14:32:41 on dist-test-slave-70rm"
server_key: "d1f7c1df53dcd1a1e5ac1ee854aaa502"
server_key_iv: "14d3df81b67b3088e6c725f1c9ea0385"
server_key_version: "encryptionkey@0"
I20260619 14:32:41.180114 10329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:41.214958 10329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:41.219326 10329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:32:41.219570 10329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:41.220340 10329 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:32:41.221529 10329 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:32:41.221608 10329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:41.221668 10329 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:32:41.221702 10329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:41.233376 10329 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.1:39135
I20260619 14:32:41.233515 10457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.1:39135 every 8 connection(s)
I20260619 14:32:41.234485 10329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
I20260619 14:32:41.238770 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 10329
I20260619 14:32:41.238924 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-0/wal/instance
I20260619 14:32:41.239236 2056 external_mini_cluster.cc:1428] Setting key fbddebf579f6fb8bcf8634c27e808f28
Jun 19 14:32:41 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879561, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.1@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.2.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.2@KRBTEST.COM" created.
I20260619 14:32:41.254973 10306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:41.236662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:41025 (local address 127.2.2.62:46287)
0619 14:32:41.236847 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:41.236852 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:41.237896 (+ 1044us) server_negotiation.cc:407] Connection header received
0619 14:32:41.240036 (+ 2140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:41.240039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:41.240085 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:41.240173 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:41.241929 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:41.242612 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:41.243799 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:41.244076 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:41.244188 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:41.247023 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:41.247029 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:41.247034 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:41.247065 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:41.251804 (+ 4739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:41.252598 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:41.252604 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:41.252608 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:41.252733 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:41.253276 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:41.253280 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:41.253281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:41.253558 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:41.253671 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:41.254562 (+ 891us) server_negotiation.cc:299] Negotiation successful
0619 14:32:41.254742 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 14:32:41.255764 10460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:41.237152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46287 (local address 127.2.2.1:41025)
0619 14:32:41.237690 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:41.237760 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:41.239804 (+ 2044us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:41.240377 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:41.240389 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:41.240972 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:41.241747 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:41.241770 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:41.242754 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:41.242758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:41.243656 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:41.243665 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:41.244177 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:41.244231 (+ 54us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:41.244414 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:41.245116 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:41.245133 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:41.246846 (+ 1713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:41.251972 (+ 5126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:41.251978 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:41.251993 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:41.252479 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:41.252887 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:41.252892 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:41.252895 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:41.253141 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:41.253753 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:41.253763 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:41.254228 (+ 465us) client_negotiation.cc:769] Sending connection context
0619 14:32:41.254479 (+ 251us) client_negotiation.cc:240] Negotiation successful
0619 14:32:41.254794 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":115,"threads_started":1}
I20260619 14:32:41.257414 10458 heartbeater.cc:347] Connected to a master server at 127.2.2.62:46287
I20260619 14:32:41.257704 10458 heartbeater.cc:464] Registering TS with master...
I20260619 14:32:41.258411 10458 heartbeater.cc:511] Master 127.2.2.62:46287 requested a full tablet report, sending...
I20260619 14:32:41.260123 10267 ts_manager.cc:194] Registered new tserver with Master: 5cb491f5b66d465e821b3f1470302c41 (127.2.2.1:39135)
I20260619 14:32:41.261300 10267 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.2.1@KRBTEST.COM'} at 127.2.2.1:41025
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab.
Entry for principal kudu/127.2.2.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab.
I20260619 14:32:41.269711 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.2:0
--local_ip_for_outbound_sockets=127.2.2.2
--webserver_interface=127.2.2.2
--webserver_port=0
--tserver_master_addrs=127.2.2.62:46287
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab
--principal=kudu/127.2.2.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:44141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:32:41.382251 10463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:41.382531 10463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:41.382606 10463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:41.387006 10463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:41.387082 10463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:41.387174 10463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.2
I20260619 14:32:41.392707 10463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:44141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab
--principal=kudu/127.2.2.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.2.2
--webserver_port=0
--tserver_master_addrs=127.2.2.62:46287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.2
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:32:41.393883 10463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:41.394979 10463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:32:41.404857 10463 server_base.cc:1034] running on GCE node
W20260619 14:32:41.404820 10468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:41.404883 10471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:41.404819 10469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:41.405601 10463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:41.406322 10463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:41.407521 10463 hybrid_clock.cc:648] HybridClock initialized: now 1781879561407504 us; error 27 us; skew 500 ppm
Jun 19 14:32:41 dist-test-slave-70rm krb5kdc[10222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879561, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:32:41.410377 10463 init.cc:377] Logged in from keytab as kudu/127.2.2.2@KRBTEST.COM (short username kudu)
I20260619 14:32:41.411357 10463 webserver.cc:460] Webserver started at http://127.2.2.2:45303/ using document root <none> and password file <none>
I20260619 14:32:41.411892 10463 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:41.411952 10463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:41.412129 10463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:32:41.414230 10463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-1/data/instance:
uuid: "190cdaf373474aeb8f07d2e05637ba9a"
format_stamp: "Formatted at 2026-06-19 14:32:41 on dist-test-slave-70rm"
server_key: "f045d15de96052f15d1e795ac7b7bfac"
server_key_iv: "80eafaed78bb922efc0cdca706885d7f"
server_key_version: "encryptionkey@0"
I20260619 14:32:41.414762 10463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-1/wal/instance:
uuid: "190cdaf373474aeb8f07d2e05637ba9a"
format_stamp: "Formatted at 2026-06-19 14:32:41 on dist-test-slave-70rm"
server_key: "f045d15de96052f15d1e795ac7b7bfac"
server_key_iv: "80eafaed78bb922efc0cdca706885d7f"
server_key_version: "encryptionkey@0"
I20260619 14:32:41.418571 10463 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260619 14:32:41.420984 10478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:41.422274 10463 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:32:41.422400 10463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-1/wal
uuid: "190cdaf373474aeb8f07d2e05637ba9a"
format_stamp: "Formatted at 2026-06-19 14:32:41 on dist-test-slave-70rm"
server_key: "f045d15de96052f15d1e795ac7b7bfac"
server_key_iv: "80eafaed78bb922efc0cdca706885d7f"
server_key_version: "encryptionkey@0"
I20260619 14:32:41.422508 10463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:41.436376 10463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:41.439421 10463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:32:41.439649 10463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:41.440308 10463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:32:41.441118 10463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:32:41.441171 10463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:41.441211 10463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:32:41.441233 10463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:41.450942 10463 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.2:33269
I20260619 14:32:41.450999 10591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.2:33269 every 8 connection(s)
I20260619 14:32:41.452036 10463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
I20260619 14:32:41.456408 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 10463
I20260619 14:32:41.456549 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-1/wal/instance
I20260619 14:32:41.456830 2056 external_mini_cluster.cc:1428] Setting key da6ffb77c34a78db77345370ed9d9586
Jun 19 14:32:41 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879561, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.2@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.2.3@KRBTEST.COM; defaulting to no policy
I20260619 14:32:41.466614 10306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:41.453976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:38747 (local address 127.2.2.62:46287)
0619 14:32:41.454107 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:41.454111 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:41.455057 (+ 946us) server_negotiation.cc:407] Connection header received
0619 14:32:41.456365 (+ 1308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:41.456368 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:41.456424 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:41.456529 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:41.457846 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:41.458497 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:41.459722 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:41.459939 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:41.460023 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:41.462717 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:41.462720 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:41.462722 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:41.462746 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:41.464285 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:41.464919 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:41.464922 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:41.464923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:41.464997 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:41.465417 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:41.465419 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:41.465420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:41.465594 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:41.465678 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:41.466382 (+ 704us) server_negotiation.cc:299] Negotiation successful
0619 14:32:41.466490 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 14:32:41.467541 10594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:41.454467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46287 (local address 127.2.2.2:38747)
0619 14:32:41.454896 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:41.454938 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:41.456156 (+ 1218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:41.456702 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:41.456710 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:41.457122 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:41.457698 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:41.457713 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:41.458621 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:41.458625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:41.459592 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:41.459605 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:41.460038 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:41.460127 (+ 89us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:41.460352 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:41.461053 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:41.461070 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:41.462534 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:41.464412 (+ 1878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:41.464420 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:41.464440 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:41.464812 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:41.465089 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:41.465092 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:41.465094 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:41.465316 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:41.465704 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:41.465711 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:41.466043 (+ 332us) client_negotiation.cc:769] Sending connection context
0619 14:32:41.466324 (+ 281us) client_negotiation.cc:240] Negotiation successful
0619 14:32:41.466643 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":110,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.3@KRBTEST.COM" created.
I20260619 14:32:41.468812 10592 heartbeater.cc:347] Connected to a master server at 127.2.2.62:46287
I20260619 14:32:41.469086 10592 heartbeater.cc:464] Registering TS with master...
I20260619 14:32:41.469682 10592 heartbeater.cc:511] Master 127.2.2.62:46287 requested a full tablet report, sending...
I20260619 14:32:41.470921 10267 ts_manager.cc:194] Registered new tserver with Master: 190cdaf373474aeb8f07d2e05637ba9a (127.2.2.2:33269)
I20260619 14:32:41.471788 10267 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.2.2@KRBTEST.COM'} at 127.2.2.2:38747
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab.
Entry for principal kudu/127.2.2.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab.
I20260619 14:32:41.484701 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.3:0
--local_ip_for_outbound_sockets=127.2.2.3
--webserver_interface=127.2.2.3
--webserver_port=0
--tserver_master_addrs=127.2.2.62:46287
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab
--principal=kudu/127.2.2.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:44141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:32:41.599800 10597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:41.600135 10597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:41.600178 10597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:41.604485 10597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:41.604555 10597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:41.604617 10597 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.3
I20260619 14:32:41.610015 10597 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:44141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab
--principal=kudu/127.2.2.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.2.3
--webserver_port=0
--tserver_master_addrs=127.2.2.62:46287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.3
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:32:41.611131 10597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:41.612128 10597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:32:41.621600 10602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:41.621615 10603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:41.621953 10605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:41.621922 10597 server_base.cc:1034] running on GCE node
I20260619 14:32:41.622452 10597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:41.623083 10597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:41.624274 10597 hybrid_clock.cc:648] HybridClock initialized: now 1781879561624257 us; error 27 us; skew 500 ppm
Jun 19 14:32:41 dist-test-slave-70rm krb5kdc[10222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879561, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:32:41.627342 10597 init.cc:377] Logged in from keytab as kudu/127.2.2.3@KRBTEST.COM (short username kudu)
I20260619 14:32:41.628226 10597 webserver.cc:460] Webserver started at http://127.2.2.3:42803/ using document root <none> and password file <none>
I20260619 14:32:41.628783 10597 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:41.628846 10597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:41.629128 10597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:32:41.631312 10597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-2/data/instance:
uuid: "c9658846c42d469091a9ef0a6d55b25a"
format_stamp: "Formatted at 2026-06-19 14:32:41 on dist-test-slave-70rm"
server_key: "f8bb54a18f0b660a72611a399ed98b31"
server_key_iv: "dd4e479cd29f2bce67768da8ce0647cc"
server_key_version: "encryptionkey@0"
I20260619 14:32:41.631917 10597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-2/wal/instance:
uuid: "c9658846c42d469091a9ef0a6d55b25a"
format_stamp: "Formatted at 2026-06-19 14:32:41 on dist-test-slave-70rm"
server_key: "f8bb54a18f0b660a72611a399ed98b31"
server_key_iv: "dd4e479cd29f2bce67768da8ce0647cc"
server_key_version: "encryptionkey@0"
I20260619 14:32:41.635638 10597 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:32:41.637960 10612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:41.639197 10597 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 14:32:41.639324 10597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-2/wal
uuid: "c9658846c42d469091a9ef0a6d55b25a"
format_stamp: "Formatted at 2026-06-19 14:32:41 on dist-test-slave-70rm"
server_key: "f8bb54a18f0b660a72611a399ed98b31"
server_key_iv: "dd4e479cd29f2bce67768da8ce0647cc"
server_key_version: "encryptionkey@0"
I20260619 14:32:41.639431 10597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:41.677946 10597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:41.681108 10597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:32:41.681324 10597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:41.681995 10597 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:32:41.682868 10597 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:32:41.682914 10597 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:41.682981 10597 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:32:41.683015 10597 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:41.693351 10597 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.3:33359
I20260619 14:32:41.693415 10725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.3:33359 every 8 connection(s)
I20260619 14:32:41.694412 10597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
I20260619 14:32:41.702186 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 10597
I20260619 14:32:41.702340 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-2/wal/instance
I20260619 14:32:41.702698 2056 external_mini_cluster.cc:1428] Setting key d2917e8ba5214c20584b3013b4f3a11b
Jun 19 14:32:41 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879561, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.3@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:32:41.708343 10306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:41.696227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.3:55047 (local address 127.2.2.62:46287)
0619 14:32:41.696372 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:41.696375 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:41.697146 (+ 771us) server_negotiation.cc:407] Connection header received
0619 14:32:41.698449 (+ 1303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:41.698452 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:41.698501 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:41.698576 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:41.699787 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:41.700413 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:41.701653 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:41.701918 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:41.702027 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:41.705080 (+ 3053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:41.705083 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:41.705084 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:41.705109 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:41.706564 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:41.707088 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:41.707091 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:41.707092 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:41.707159 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:41.707460 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:41.707462 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:41.707463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:41.707625 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:41.707696 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:41.708133 (+ 437us) server_negotiation.cc:299] Negotiation successful
0619 14:32:41.708226 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 14:32:41.709188 10728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:41.696553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46287 (local address 127.2.2.3:55047)
0619 14:32:41.696990 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:41.697032 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:41.698244 (+ 1212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:41.698758 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:41.698765 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:41.699145 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:41.699639 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:41.699654 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:41.700548 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:41.700551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:41.701552 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:41.701561 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:41.702012 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:41.702063 (+ 51us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:41.702262 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:41.702961 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:41.702978 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:41.704910 (+ 1932us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:41.706659 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:41.706664 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:41.706677 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:41.707003 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:41.707252 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:32:41.707254 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:32:41.707256 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:32:41.707385 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:32:41.707707 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:41.707712 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:41.707938 (+ 226us) client_negotiation.cc:769] Sending connection context
0619 14:32:41.708093 (+ 155us) client_negotiation.cc:240] Negotiation successful
0619 14:32:41.708366 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":126,"threads_started":1}
I20260619 14:32:41.710458 10726 heartbeater.cc:347] Connected to a master server at 127.2.2.62:46287
I20260619 14:32:41.710676 10726 heartbeater.cc:464] Registering TS with master...
I20260619 14:32:41.711126 10726 heartbeater.cc:511] Master 127.2.2.62:46287 requested a full tablet report, sending...
I20260619 14:32:41.712096 10267 ts_manager.cc:194] Registered new tserver with Master: c9658846c42d469091a9ef0a6d55b25a (127.2.2.3:33359)
I20260619 14:32:41.712613 10267 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.2.3@KRBTEST.COM'} at 127.2.2.3:55047
I20260619 14:32:41.716800 2056 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.2.2.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.2.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.2.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/krb5kdc/oryx_127.2.2.1.keytab.
Entry for principal oryx/127.2.2.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/krb5kdc/oryx_127.2.2.1.keytab.
WARNING: no policy specified for oryx/127.2.2.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.2.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.2.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/krb5kdc/oryx_127.2.2.2.keytab.
Entry for principal oryx/127.2.2.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/krb5kdc/oryx_127.2.2.2.keytab.
WARNING: no policy specified for oryx/127.2.2.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.2.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.2.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/krb5kdc/oryx_127.2.2.3.keytab.
Entry for principal oryx/127.2.2.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/krb5kdc/oryx_127.2.2.3.keytab.
I20260619 14:32:41.788477 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 10329
I20260619 14:32:41.793664 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 10463
I20260619 14:32:41.798956 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 10597
I20260619 14:32:41.804363 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 10235
I20260619 14:32:41.810714 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:46287
--webserver_interface=127.2.2.62
--webserver_port=43833
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:44141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.2.62:46287
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:32:41.920921 10735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:41.921203 10735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:41.921245 10735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:41.925304 10735 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:32:41.925377 10735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:41.925396 10735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:41.925413 10735 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:32:41.925428 10735 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:32:41.930866 10735 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:44141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.2.62:46287
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:46287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.2.62
--webserver_port=43833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:32:41.931896 10735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:41.932861 10735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:32:41.941648 10735 server_base.cc:1034] running on GCE node
W20260619 14:32:41.941597 10743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:41.941596 10741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:41.941596 10740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:41.942504 10735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:41.943658 10735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:41.944849 10735 hybrid_clock.cc:648] HybridClock initialized: now 1781879561944825 us; error 38 us; skew 500 ppm
Jun 19 14:32:41 dist-test-slave-70rm krb5kdc[10222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879561, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:32:41.947625 10735 init.cc:377] Logged in from keytab as kudu/127.2.2.62@KRBTEST.COM (short username kudu)
I20260619 14:32:41.948460 10735 webserver.cc:460] Webserver started at http://127.2.2.62:43833/ using document root <none> and password file <none>
I20260619 14:32:41.948980 10735 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:41.949049 10735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:41.952937 10735 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:32:41.955205 10750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:41.956486 10735 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:32:41.956609 10735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/master-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/master-0/wal
uuid: "68cfe4990e2240f8a1936a97a7c6139c"
format_stamp: "Formatted at 2026-06-19 14:32:40 on dist-test-slave-70rm"
server_key: "b59f7718a0c8bd35c38d729f7585b661"
server_key_iv: "64665485422962e9448454b7361c676a"
server_key_version: "encryptionkey@0"
I20260619 14:32:41.957125 10735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:41.972385 10735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:41.975520 10735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:32:41.975744 10735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:41.982708 10735 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.62:46287
I20260619 14:32:41.982779 10802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.62:46287 every 8 connection(s)
I20260619 14:32:41.983827 10735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
I20260619 14:32:41.986871 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 10735
I20260619 14:32:41.987294 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.1:39135
--local_ip_for_outbound_sockets=127.2.2.1
--tserver_master_addrs=127.2.2.62:46287
--webserver_port=35585
--webserver_interface=127.2.2.1
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab
--principal=kudu/127.2.2.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:44141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.2.1
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/krb5kdc/oryx_127.2.2.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:32:41.988180 10803 sys_catalog.cc:263] Verifying existing consensus state
I20260619 14:32:41.990525 10803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68cfe4990e2240f8a1936a97a7c6139c: Bootstrap starting.
I20260619 14:32:42.007499 10803 log.cc:824] T 00000000000000000000000000000000 P 68cfe4990e2240f8a1936a97a7c6139c: Log is configured to *not* fsync() on all Append() calls
I20260619 14:32:42.014227 10803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68cfe4990e2240f8a1936a97a7c6139c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:32:42.014623 10803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68cfe4990e2240f8a1936a97a7c6139c: Bootstrap complete.
I20260619 14:32:42.018915 10803 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68cfe4990e2240f8a1936a97a7c6139c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68cfe4990e2240f8a1936a97a7c6139c" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 46287 } }
I20260619 14:32:42.019352 10803 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68cfe4990e2240f8a1936a97a7c6139c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68cfe4990e2240f8a1936a97a7c6139c, State: Initialized, Role: FOLLOWER
I20260619 14:32:42.019891 10803 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68cfe4990e2240f8a1936a97a7c6139c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68cfe4990e2240f8a1936a97a7c6139c" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 46287 } }
I20260619 14:32:42.020037 10803 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68cfe4990e2240f8a1936a97a7c6139c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:32:42.020125 10803 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68cfe4990e2240f8a1936a97a7c6139c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:32:42.020238 10803 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68cfe4990e2240f8a1936a97a7c6139c [term 1 FOLLOWER]: Advancing to term 2
I20260619 14:32:42.022367 10803 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68cfe4990e2240f8a1936a97a7c6139c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68cfe4990e2240f8a1936a97a7c6139c" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 46287 } }
I20260619 14:32:42.022702 10803 leader_election.cc:302] T 00000000000000000000000000000000 P 68cfe4990e2240f8a1936a97a7c6139c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68cfe4990e2240f8a1936a97a7c6139c; no voters:
I20260619 14:32:42.023042 10803 leader_election.cc:288] T 00000000000000000000000000000000 P 68cfe4990e2240f8a1936a97a7c6139c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 14:32:42.023140 10807 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68cfe4990e2240f8a1936a97a7c6139c [term 2 FOLLOWER]: Leader election won for term 2
I20260619 14:32:42.023334 10807 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68cfe4990e2240f8a1936a97a7c6139c [term 2 LEADER]: Becoming Leader. State: Replica: 68cfe4990e2240f8a1936a97a7c6139c, State: Running, Role: LEADER
I20260619 14:32:42.023633 10807 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68cfe4990e2240f8a1936a97a7c6139c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68cfe4990e2240f8a1936a97a7c6139c" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 46287 } }
I20260619 14:32:42.024307 10803 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68cfe4990e2240f8a1936a97a7c6139c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:32:42.024612 10809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68cfe4990e2240f8a1936a97a7c6139c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68cfe4990e2240f8a1936a97a7c6139c. Latest consensus state: current_term: 2 leader_uuid: "68cfe4990e2240f8a1936a97a7c6139c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68cfe4990e2240f8a1936a97a7c6139c" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 46287 } } }
I20260619 14:32:42.024737 10809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68cfe4990e2240f8a1936a97a7c6139c [sys.catalog]: This master's current role is: LEADER
I20260619 14:32:42.024957 10808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68cfe4990e2240f8a1936a97a7c6139c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "68cfe4990e2240f8a1936a97a7c6139c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68cfe4990e2240f8a1936a97a7c6139c" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 46287 } } }
I20260619 14:32:42.025060 10808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68cfe4990e2240f8a1936a97a7c6139c [sys.catalog]: This master's current role is: LEADER
I20260619 14:32:42.025110 10811 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:32:42.028429 10811 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:32:42.029067 10811 catalog_manager.cc:1259] Loaded cluster ID: e43c144fac7e4be292d6ddc814ae0217
I20260619 14:32:42.029130 10811 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:32:42.031603 10811 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:32:42.032290 10811 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 68cfe4990e2240f8a1936a97a7c6139c: Loaded TSK: 0
I20260619 14:32:42.033553 10811 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 14:32:42.105537 10805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:42.105852 10805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:42.105945 10805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:42.110193 10805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:42.110311 10805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:42.110410 10805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.1
I20260619 14:32:42.116204 10805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:44141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/krb5kdc/oryx_127.2.2.1.keytab
--principal=oryx/127.2.2.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.1:39135
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.2.1
--webserver_port=35585
--tserver_master_addrs=127.2.2.62:46287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.1
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:32:42.117337 10805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:42.118470 10805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:32:42.128468 10829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:42.128513 10832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:42.128949 10830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:42.129344 10805 server_base.cc:1034] running on GCE node
I20260619 14:32:42.129798 10805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:42.130532 10805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:42.131742 10805 hybrid_clock.cc:648] HybridClock initialized: now 1781879562131721 us; error 70 us; skew 500 ppm
Jun 19 14:32:42 dist-test-slave-70rm krb5kdc[10222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879562, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.2.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:32:42.134749 10805 init.cc:377] Logged in from keytab as oryx/127.2.2.1@KRBTEST.COM (short username oryx)
I20260619 14:32:42.135743 10805 webserver.cc:460] Webserver started at http://127.2.2.1:35585/ using document root <none> and password file <none>
I20260619 14:32:42.136323 10805 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:42.136400 10805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:42.140556 10805 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:32:42.142765 10839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:42.144075 10805 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:32:42.144170 10805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-0/wal
uuid: "5cb491f5b66d465e821b3f1470302c41"
format_stamp: "Formatted at 2026-06-19 14:32:41 on dist-test-slave-70rm"
server_key: "d1f7c1df53dcd1a1e5ac1ee854aaa502"
server_key_iv: "14d3df81b67b3088e6c725f1c9ea0385"
server_key_version: "encryptionkey@0"
I20260619 14:32:42.144747 10805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:42.166694 10805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:42.170352 10805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:32:42.170571 10805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:42.171253 10805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:32:42.172135 10805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:32:42.172186 10805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:42.172254 10805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:32:42.172286 10805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:42.181684 10805 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.1:39135
I20260619 14:32:42.181694 10952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.1:39135 every 8 connection(s)
I20260619 14:32:42.182754 10805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
I20260619 14:32:42.184281 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 10805
I20260619 14:32:42.184798 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.2:33269
--local_ip_for_outbound_sockets=127.2.2.2
--tserver_master_addrs=127.2.2.62:46287
--webserver_port=45303
--webserver_interface=127.2.2.2
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab
--principal=kudu/127.2.2.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:44141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.2.2
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/krb5kdc/oryx_127.2.2.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:32:42 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.1@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:32:42 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.1@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 14:32:42.196892 10955 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:32:42.185166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:43473 (local address 127.2.2.62:46287)
0619 14:32:42.185717 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:42.185739 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:42.187425 (+ 1686us) server_negotiation.cc:407] Connection header received
0619 14:32:42.187653 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:42.187663 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:42.187959 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:42.188294 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:42.189462 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:42.190551 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:42.191778 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:42.192273 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:42.192455 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:42.196198 (+ 3743us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.2.1:43473: BlockingRecv error: recv got EOF from 127.2.2.1:43473 (error 108)
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":134,"threads_started":1}
I20260619 14:32:42.196936 10956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:42.185283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46287 (local address 127.2.2.1:43473)
0619 14:32:42.185875 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:42.185928 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:42.187171 (+ 1243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:42.188357 (+ 1186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:42.188365 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:42.188804 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:42.189284 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:42.189297 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:42.190713 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:42.190716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:42.191643 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:42.191653 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:42.192386 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:42.192447 (+ 61us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:42.192636 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:42.193284 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:42.193301 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:42.195979 (+ 2678us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.2.62:46287: Server oryx/127.2.2.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":138,"threads_started":1}
W20260619 14:32:42.197383 10953 heartbeater.cc:644] Failed to heartbeat to 127.2.2.62:46287 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.2.62:46287: Client connection negotiation failed: client connection to 127.2.2.62:46287: Server oryx/127.2.2.62@KRBTEST.COM not found in Kerberos database
Jun 19 14:32:42 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.1@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:32:42 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.1@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:32:42.203917 10956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:42.197847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46287 (local address 127.2.2.1:60981)
0619 14:32:42.197966 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:42.197982 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:42.198068 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:42.198438 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:42.198441 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:42.198650 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:42.198770 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:42.198777 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:42.199505 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:42.199510 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:42.200405 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:42.200414 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:42.200874 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:42.200891 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:42.200984 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:42.201612 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:42.201633 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:42.203774 (+ 2141us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.2.62:46287: Server oryx/127.2.2.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 14:32:42.204231 10955 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:32:42.198057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:60981 (local address 127.2.2.62:46287)
0619 14:32:42.198231 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:42.198234 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:42.198247 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:32:42.198284 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:42.198286 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:42.198326 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:42.198394 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:42.198888 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:42.199404 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:42.200550 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:42.200776 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:42.200866 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:42.203937 (+ 3071us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.2.1:60981: BlockingRecv error: recv got EOF from 127.2.2.1:60981 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Jun 19 14:32:42 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.1@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:32:42 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.1@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:32:42.209880 10956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:42.204540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46287 (local address 127.2.2.1:52247)
0619 14:32:42.204702 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:42.204716 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:42.204783 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:42.205009 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:42.205012 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:42.205172 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:42.205258 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:42.205263 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:42.205936 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:42.205941 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:42.206830 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:42.206839 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:42.207292 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:42.207307 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:42.207380 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:42.207875 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:42.207889 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:42.209759 (+ 1870us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.2.62:46287: Server oryx/127.2.2.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102}
W20260619 14:32:42.210093 10955 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:32:42.204639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:52247 (local address 127.2.2.62:46287)
0619 14:32:42.204745 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:42.204751 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:42.204770 (+ 19us) server_negotiation.cc:407] Connection header received
0619 14:32:42.204868 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:42.204870 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:42.204907 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:42.204974 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:42.205359 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:42.205836 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:42.206959 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:42.207189 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:42.207277 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:42.209918 (+ 2641us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.2.1:52247: BlockingRecv error: recv got EOF from 127.2.2.1:52247 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260619 14:32:42.210218 10953 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 14:32:42.303467 10957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:42.303800 10957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:42.303879 10957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:42.308084 10957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:42.308187 10957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:42.308282 10957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.2
I20260619 14:32:42.313656 10957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:44141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/krb5kdc/oryx_127.2.2.2.keytab
--principal=oryx/127.2.2.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.2:33269
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.2.2
--webserver_port=45303
--tserver_master_addrs=127.2.2.62:46287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.2
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:32:42.314896 10957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:42.315984 10957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:32:42.326216 10965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:42.326239 10963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:42.326208 10962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:42.326285 10957 server_base.cc:1034] running on GCE node
I20260619 14:32:42.326845 10957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:42.327545 10957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:42.328756 10957 hybrid_clock.cc:648] HybridClock initialized: now 1781879562328745 us; error 35 us; skew 500 ppm
Jun 19 14:32:42 dist-test-slave-70rm krb5kdc[10222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879562, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.2.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:32:42.331799 10957 init.cc:377] Logged in from keytab as oryx/127.2.2.2@KRBTEST.COM (short username oryx)
I20260619 14:32:42.332736 10957 webserver.cc:460] Webserver started at http://127.2.2.2:45303/ using document root <none> and password file <none>
I20260619 14:32:42.333364 10957 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:42.333483 10957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:42.337595 10957 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:32:42.339740 10972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:42.340946 10957 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:32:42.341117 10957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-1/wal
uuid: "190cdaf373474aeb8f07d2e05637ba9a"
format_stamp: "Formatted at 2026-06-19 14:32:41 on dist-test-slave-70rm"
server_key: "f045d15de96052f15d1e795ac7b7bfac"
server_key_iv: "80eafaed78bb922efc0cdca706885d7f"
server_key_version: "encryptionkey@0"
I20260619 14:32:42.341686 10957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:42.365520 10957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:42.369099 10957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:32:42.369315 10957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:42.369925 10957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:32:42.370867 10957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:32:42.370936 10957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:42.371007 10957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:32:42.371049 10957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:42.380435 10957 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.2:33269
I20260619 14:32:42.380460 11085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.2:33269 every 8 connection(s)
I20260619 14:32:42.381521 10957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
I20260619 14:32:42.381958 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 10957
I20260619 14:32:42.382458 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.3:33359
--local_ip_for_outbound_sockets=127.2.2.3
--tserver_master_addrs=127.2.2.62:46287
--webserver_port=42803
--webserver_interface=127.2.2.3
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab
--principal=kudu/127.2.2.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:44141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.2.3
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/krb5kdc/oryx_127.2.2.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:32:42 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.2@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:32:42 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.2@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 14:32:42.394488 10955 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:32:42.383749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:37027 (local address 127.2.2.62:46287)
0619 14:32:42.383938 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:42.383945 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:42.384918 (+ 973us) server_negotiation.cc:407] Connection header received
0619 14:32:42.386476 (+ 1558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:42.386482 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:42.386564 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:42.386689 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:42.388001 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:42.388938 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:42.390173 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:42.390608 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:42.390732 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:42.394352 (+ 3620us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.2.2:37027: BlockingRecv error: recv got EOF from 127.2.2.2:37027 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 14:32:42.395104 11089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:42.384218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46287 (local address 127.2.2.2:37027)
0619 14:32:42.384720 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:42.384777 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:42.386241 (+ 1464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:42.386896 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:42.386904 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:42.387303 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:42.387816 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:42.387830 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:42.389100 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:42.389103 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:42.390018 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:42.390027 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:42.390732 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:42.390791 (+ 59us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:42.390955 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:42.391676 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:42.391694 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:42.394210 (+ 2516us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.2.62:46287: Server oryx/127.2.2.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":157,"threads_started":1}
W20260619 14:32:42.395857 11086 heartbeater.cc:644] Failed to heartbeat to 127.2.2.62:46287 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.2.62:46287: Client connection negotiation failed: client connection to 127.2.2.62:46287: Server oryx/127.2.2.62@KRBTEST.COM not found in Kerberos database
Jun 19 14:32:42 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.2@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:32:42 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.2@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:32:42.401929 11089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:42.396305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46287 (local address 127.2.2.2:44813)
0619 14:32:42.396443 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:42.396458 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:42.396543 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:42.396759 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:42.396762 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:42.396938 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:42.397055 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:42.397060 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:42.397756 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:42.397759 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:42.398720 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:42.398735 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:42.399201 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:42.399219 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:42.399324 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:42.399800 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:42.399815 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:42.401791 (+ 1976us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.2.62:46287: Server oryx/127.2.2.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":58}
W20260619 14:32:42.402313 10955 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:32:42.396425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:44813 (local address 127.2.2.62:46287)
0619 14:32:42.396551 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:42.396555 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:42.396568 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:32:42.396618 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:42.396620 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:42.396657 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:42.396724 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:42.397168 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:42.397660 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:42.398871 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:42.399108 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:42.399187 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:42.402178 (+ 2991us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.2.2:44813: BlockingRecv error: recv got EOF from 127.2.2.2:44813 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 19 14:32:42 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.2@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:32:42 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.2@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:32:42.407871 11089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:42.402643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46287 (local address 127.2.2.2:36719)
0619 14:32:42.402778 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:42.402794 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:42.402874 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:42.403119 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:42.403122 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:42.403290 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:42.403381 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:42.403387 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:42.404145 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:42.404148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:42.404913 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:42.404924 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:42.405377 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:42.405394 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:42.405467 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:42.405938 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:42.405951 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:42.407773 (+ 1822us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.2.62:46287: Server oryx/127.2.2.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260619 14:32:42.408046 10955 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:32:42.402736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:36719 (local address 127.2.2.62:46287)
0619 14:32:42.402846 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:42.402850 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:42.402864 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:32:42.402979 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:42.402982 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:42.403019 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:42.403089 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:42.403532 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:42.404036 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:42.405043 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:42.405270 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:42.405360 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:42.407930 (+ 2570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.2.2:36719: BlockingRecv error: recv got EOF from 127.2.2.2:36719 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260619 14:32:42.408358 11086 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 14:32:42.496465 11088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:42.496948 11088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:42.497078 11088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:42.501277 11088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:42.501341 11088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:42.501441 11088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.3
I20260619 14:32:42.507000 11088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:44141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/krb5kdc/oryx_127.2.2.3.keytab
--principal=oryx/127.2.2.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.3:33359
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.2.3
--webserver_port=42803
--tserver_master_addrs=127.2.2.62:46287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.3
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:32:42.508111 11088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:42.509248 11088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:32:42.518836 11097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:42.518841 11094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:42.519143 11095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:42.519366 11088 server_base.cc:1034] running on GCE node
I20260619 14:32:42.519729 11088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:42.520356 11088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:42.521513 11088 hybrid_clock.cc:648] HybridClock initialized: now 1781879562521499 us; error 33 us; skew 500 ppm
Jun 19 14:32:42 dist-test-slave-70rm krb5kdc[10222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879562, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.2.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:32:42.524233 11088 init.cc:377] Logged in from keytab as oryx/127.2.2.3@KRBTEST.COM (short username oryx)
I20260619 14:32:42.525324 11088 webserver.cc:460] Webserver started at http://127.2.2.3:42803/ using document root <none> and password file <none>
I20260619 14:32:42.525935 11088 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:42.526046 11088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:42.530239 11088 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:32:42.532503 11104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:42.533938 11088 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:32:42.534088 11088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-2/wal
uuid: "c9658846c42d469091a9ef0a6d55b25a"
format_stamp: "Formatted at 2026-06-19 14:32:41 on dist-test-slave-70rm"
server_key: "f8bb54a18f0b660a72611a399ed98b31"
server_key_iv: "dd4e479cd29f2bce67768da8ce0647cc"
server_key_version: "encryptionkey@0"
I20260619 14:32:42.534741 11088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:42.547221 11088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:42.550551 11088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:32:42.550761 11088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:42.551373 11088 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:32:42.552215 11088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:32:42.552263 11088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:42.552350 11088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:32:42.552393 11088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:42.561713 11088 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.3:33359
I20260619 14:32:42.561774 11217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.3:33359 every 8 connection(s)
I20260619 14:32:42.562808 11088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
I20260619 14:32:42.569236 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 11088
Jun 19 14:32:42 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.3@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:32:42 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.3@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 14:32:42.574470 10955 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:32:42.564704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.3:35005 (local address 127.2.2.62:46287)
0619 14:32:42.564839 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:42.564843 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:42.565713 (+ 870us) server_negotiation.cc:407] Connection header received
0619 14:32:42.567058 (+ 1345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:42.567062 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:42.567120 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:42.567200 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:42.568371 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:42.568899 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:42.570405 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:42.570700 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:42.570849 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:42.574313 (+ 3464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.2.3:35005: BlockingRecv error: recv got EOF from 127.2.2.3:35005 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 14:32:42.575232 11220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:42.565130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46287 (local address 127.2.2.3:35005)
0619 14:32:42.565557 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:42.565600 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:42.566839 (+ 1239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:42.567373 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:42.567380 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:42.567736 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:42.568219 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:42.568232 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:42.569030 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:42.569034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:42.570276 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:42.570288 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:42.570841 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:42.570897 (+ 56us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:42.571044 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:42.571705 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:42.571721 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:42.574101 (+ 2380us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.2.62:46287: Server oryx/127.2.2.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":119,"threads_started":1}
W20260619 14:32:42.577102 11218 heartbeater.cc:644] Failed to heartbeat to 127.2.2.62:46287 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.2.62:46287: Client connection negotiation failed: client connection to 127.2.2.62:46287: Server oryx/127.2.2.62@KRBTEST.COM not found in Kerberos database
I20260619 14:32:42.580442 11222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:42.570344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58008 (local address 127.2.2.62:46287)
0619 14:32:42.570592 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:42.570595 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:42.570606 (+ 11us) server_negotiation.cc:407] Connection header received
0619 14:32:42.570637 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:42.570639 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:42.570674 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:42.570746 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:42.571591 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:42.572113 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:42.573020 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:42.573265 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:42.573387 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:42.575182 (+ 1795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:42.575188 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:42.575208 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:32:42.575244 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:42.577747 (+ 2503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:42.578432 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:42.578441 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:42.578448 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:42.578630 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:32:42.579078 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:32:42.579082 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:32:42.579084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:32:42.579453 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:42.579681 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:42.579974 (+ 293us) server_negotiation.cc:299] Negotiation successful
0619 14:32:42.580277 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":93,"threads_started":1}
Jun 19 14:32:42 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.3@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:32:42 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.3@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:32:42.583938 11220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:42.577605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46287 (local address 127.2.2.3:52879)
0619 14:32:42.578371 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:42.578387 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:42.578478 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:42.578850 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:42.578854 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:42.578999 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:42.579100 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:42.579105 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:42.579845 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:42.579849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:42.580609 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:42.580619 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:42.581139 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:42.581153 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:42.581223 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:42.581717 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:42.581731 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:42.583842 (+ 2111us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.2.62:46287: Server oryx/127.2.2.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":633}
W20260619 14:32:42.584089 10955 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:32:42.577679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.3:52879 (local address 127.2.2.62:46287)
0619 14:32:42.577848 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:42.577852 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:42.578561 (+ 709us) server_negotiation.cc:407] Connection header received
0619 14:32:42.578604 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:42.578608 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:42.578657 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:42.578740 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:42.579217 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:42.579729 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:42.580723 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:42.581028 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:42.581123 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:42.583942 (+ 2819us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.2.3:52879: BlockingRecv error: recv got EOF from 127.2.2.3:52879 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 19 14:32:42 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.3@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:32:42 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.3@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:32:42.589284 11220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:42.584430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46287 (local address 127.2.2.3:58929)
0619 14:32:42.584539 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:42.584554 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:42.584626 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:42.584817 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:42.584819 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:42.584939 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:42.585009 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:42.585014 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:42.585716 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:42.585719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:42.586520 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:42.586529 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:42.586917 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:42.586930 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:42.586986 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:42.587433 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:42.587444 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:42.589182 (+ 1738us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.2.62:46287: Server oryx/127.2.2.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 14:32:42.589417 10955 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:32:42.584490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.3:58929 (local address 127.2.2.62:46287)
0619 14:32:42.584623 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:42.584626 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:42.584637 (+ 11us) server_negotiation.cc:407] Connection header received
0619 14:32:42.584687 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:42.584689 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:42.584721 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:42.584783 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:42.585101 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:42.585604 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:42.586627 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:42.586830 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:42.586911 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:42.589305 (+ 2394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.2.3:58929: BlockingRecv error: recv got EOF from 127.2.2.3:58929 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260619 14:32:42.589510 11218 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 19 14:32:43 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.1@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:32:43 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.1@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:32:43.216605 11223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:43.210928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46287 (local address 127.2.2.1:58281)
0619 14:32:43.211221 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:43.211245 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:43.211342 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:43.211573 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:43.211575 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:43.211753 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:43.211847 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:43.211852 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:43.212549 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:43.212551 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:43.213146 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:43.213154 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:43.213587 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:43.213605 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:43.213675 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:43.214324 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:43.214346 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:43.216475 (+ 2129us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.2.62:46287: Server oryx/127.2.2.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":68,"threads_started":1}
W20260619 14:32:43.216768 11224 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:32:43.211077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:58281 (local address 127.2.2.62:46287)
0619 14:32:43.211313 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:43.211317 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:43.211331 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:32:43.211413 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:43.211415 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:43.211456 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:43.211532 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:43.211942 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:43.212453 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:43.213251 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:43.213477 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:43.213558 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:43.216630 (+ 3072us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.2.1:58281: BlockingRecv error: recv got EOF from 127.2.2.1:58281 (error 108)
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":74,"threads_started":1}
Jun 19 14:32:43 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.2@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:32:43 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.2@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:32:43.414324 11225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:43.408994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46287 (local address 127.2.2.2:54319)
0619 14:32:43.409261 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:43.409275 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:43.409373 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:43.409602 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:43.409604 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:43.409792 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:43.409884 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:43.409889 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:43.410836 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:43.410838 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:43.411405 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:43.411413 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:43.411807 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:43.411822 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:43.411881 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:43.412397 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:43.412408 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:43.414196 (+ 1788us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.2.62:46287: Server oryx/127.2.2.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":98,"threads_started":1}
W20260619 14:32:43.414501 11224 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:32:43.409126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:54319 (local address 127.2.2.62:46287)
0619 14:32:43.409250 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:43.409254 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:43.409328 (+ 74us) server_negotiation.cc:407] Connection header received
0619 14:32:43.409443 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:43.409445 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:43.409486 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:43.409557 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:43.410037 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:43.410712 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:43.411503 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:43.411724 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:43.411803 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:43.414369 (+ 2566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.2.2:54319: BlockingRecv error: recv got EOF from 127.2.2.2:54319 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Jun 19 14:32:43 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.3@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:32:43 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.3@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:32:43.599483 11226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:43.593741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46287 (local address 127.2.2.3:54585)
0619 14:32:43.594000 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:43.594015 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:43.594161 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:43.594458 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:43.594461 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:43.594675 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:43.594796 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:43.594803 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:43.595852 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:43.595854 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:43.596420 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:43.596426 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:43.596837 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:43.596852 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:43.596915 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:43.597407 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:43.597419 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:43.599362 (+ 1943us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.2.62:46287: Server oryx/127.2.2.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":98,"threads_started":1}
W20260619 14:32:43.599717 11224 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:32:43.593799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.3:54585 (local address 127.2.2.62:46287)
0619 14:32:43.593975 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:43.593981 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:43.594086 (+ 105us) server_negotiation.cc:407] Connection header received
0619 14:32:43.594288 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:43.594291 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:43.594348 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:43.594434 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:43.594924 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:43.595695 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:43.596516 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:43.596746 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:43.596828 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:43.599496 (+ 2668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.2.3:54585: BlockingRecv error: recv got EOF from 127.2.2.3:54585 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 19 14:32:44 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.1@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:32:44 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.1@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:32:44.223682 11227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:44.217580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46287 (local address 127.2.2.1:55821)
0619 14:32:44.217869 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:44.217882 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:44.217974 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:44.218388 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:44.218391 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:44.218568 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:44.218661 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:44.218666 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:44.219729 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:44.219733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:44.220552 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:44.220564 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:44.221027 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:44.221043 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:44.221107 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:44.221633 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:44.221644 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:44.223565 (+ 1921us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.2.62:46287: Server oryx/127.2.2.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":102,"threads_started":1}
W20260619 14:32:44.223909 11228 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:32:44.217755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:55821 (local address 127.2.2.62:46287)
0619 14:32:44.218143 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:44.218147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:44.218164 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:32:44.218210 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:44.218212 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:44.218256 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:44.218356 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:44.218809 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:44.219575 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:44.220690 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:44.220928 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:44.221017 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:44.223732 (+ 2715us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.2.1:55821: BlockingRecv error: recv got EOF from 127.2.2.1:55821 (error 108)
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":187,"threads_started":1}
Jun 19 14:32:44 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.2@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:32:44 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.2@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:32:44.421053 11229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:44.415130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46287 (local address 127.2.2.2:37263)
0619 14:32:44.415484 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:44.415501 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:44.415652 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:44.415936 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:44.415939 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:44.416182 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:44.416274 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:44.416279 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:44.417045 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:44.417051 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:44.417624 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:44.417631 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:44.418078 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:44.418098 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:44.418229 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:44.418792 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:44.418804 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:44.420854 (+ 2050us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.2.62:46287: Server oryx/127.2.2.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":112,"threads_started":1}
W20260619 14:32:44.421128 11228 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:32:44.415285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:37263 (local address 127.2.2.62:46287)
0619 14:32:44.415465 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:44.415470 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:44.415594 (+ 124us) server_negotiation.cc:407] Connection header received
0619 14:32:44.415769 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:44.415773 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:44.415825 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:44.415914 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:44.416397 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:44.416933 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:44.417756 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:44.417981 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:44.418088 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:44.421007 (+ 2919us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.2.2:37263: BlockingRecv error: recv got EOF from 127.2.2.2:37263 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Jun 19 14:32:44 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.3@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:32:44 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.3@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:32:44.605964 11230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:44.600319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46287 (local address 127.2.2.3:57449)
0619 14:32:44.600592 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:44.600610 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:44.600733 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:44.600975 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:44.600978 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:44.601173 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:44.601268 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:44.601273 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:44.601993 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:44.601996 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:44.602648 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:44.602658 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:44.603076 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:44.603111 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:44.603175 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:44.603757 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:44.603770 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:44.605842 (+ 2072us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.2.62:46287: Server oryx/127.2.2.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":81,"threads_started":1}
W20260619 14:32:44.606276 11228 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:32:44.600457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.3:57449 (local address 127.2.2.62:46287)
0619 14:32:44.600593 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:44.600599 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:44.600694 (+ 95us) server_negotiation.cc:407] Connection header received
0619 14:32:44.600827 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:44.600829 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:44.600874 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:44.600941 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:44.601365 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:44.601883 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:44.602763 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:44.602986 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:44.603062 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:44.605992 (+ 2930us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.2.3:57449: BlockingRecv error: recv got EOF from 127.2.2.3:57449 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 19 14:32:45 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.1@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:32:45 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.1@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:32:45.230386 11231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:45.224508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46287 (local address 127.2.2.1:36507)
0619 14:32:45.224851 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:45.224865 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:45.224957 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:45.225214 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:45.225217 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:45.225464 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:45.225661 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:45.225667 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:45.226444 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:45.226446 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:45.227070 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:45.227078 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:45.227528 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:45.227545 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:45.227603 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:45.228106 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:45.228117 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:45.230202 (+ 2085us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.2.62:46287: Server oryx/127.2.2.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":121,"threads_started":1}
W20260619 14:32:45.230506 11232 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:32:45.224645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:36507 (local address 127.2.2.62:46287)
0619 14:32:45.224918 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:45.224921 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:45.224937 (+ 16us) server_negotiation.cc:407] Connection header received
0619 14:32:45.225047 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:45.225049 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:45.225097 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:45.225177 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:45.225786 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:45.226360 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:45.227213 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:45.227439 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:45.227528 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:45.230358 (+ 2830us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.2.1:36507: BlockingRecv error: recv got EOF from 127.2.2.1:36507 (error 108)
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":89,"threads_started":1}
Jun 19 14:32:45 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.2@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:32:45 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.2@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:32:45.427556 11233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:45.421888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46287 (local address 127.2.2.2:53209)
0619 14:32:45.422223 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:45.422236 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:45.422334 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:45.422538 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:45.422540 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:45.422721 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:45.422812 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:45.422816 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:45.423553 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:45.423556 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:45.424246 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:45.424256 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:45.424681 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:45.424697 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:45.424776 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:45.425429 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:45.425447 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:45.427420 (+ 1973us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.2.62:46287: Server oryx/127.2.2.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":96,"threads_started":1}
W20260619 14:32:45.427695 11232 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:32:45.421958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:53209 (local address 127.2.2.62:46287)
0619 14:32:45.422106 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:45.422113 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:45.422311 (+ 198us) server_negotiation.cc:407] Connection header received
0619 14:32:45.422402 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:45.422404 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:45.422447 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:45.422516 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:45.422907 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:45.423430 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:45.424363 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:45.424577 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:45.424654 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:45.427566 (+ 2912us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.2.2:53209: BlockingRecv error: recv got EOF from 127.2.2.2:53209 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 19 14:32:45 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.3@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:32:45 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.3@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:32:45.612907 11234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:45.606808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46287 (local address 127.2.2.3:50231)
0619 14:32:45.607119 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:45.607133 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:45.607238 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:45.607508 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:45.607511 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:45.607693 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:45.607793 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:45.607798 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:45.608546 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:45.608549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:45.609133 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:45.609143 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:45.609576 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:45.609597 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:45.609661 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:45.610483 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:45.610495 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:45.612713 (+ 2218us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.2.62:46287: Server oryx/127.2.2.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":113,"threads_started":1}
W20260619 14:32:45.612938 11232 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:32:45.606890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.3:50231 (local address 127.2.2.62:46287)
0619 14:32:45.607015 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:45.607018 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:45.607202 (+ 184us) server_negotiation.cc:407] Connection header received
0619 14:32:45.607320 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:45.607322 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:45.607369 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:45.607447 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:45.607907 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:45.608438 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:45.609268 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:45.609482 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:45.609567 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:45.612837 (+ 3270us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.2.3:50231: BlockingRecv error: recv got EOF from 127.2.2.3:50231 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Jun 19 14:32:46 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.1@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:32:46 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.1@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:32:46.243358 11235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:46.231501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46287 (local address 127.2.2.1:39499)
0619 14:32:46.231781 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:46.231793 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:46.231889 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:46.232285 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:46.232287 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:46.232470 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:46.232567 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:46.232572 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:46.233550 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:46.233553 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:46.234349 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:46.234363 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:46.234995 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:46.235015 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:46.235115 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:46.240967 (+ 5852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:46.240983 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:46.243221 (+ 2238us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.2.62:46287: Server oryx/127.2.2.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":91,"threads_started":1}
W20260619 14:32:46.243599 11236 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:32:46.231590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:39499 (local address 127.2.2.62:46287)
0619 14:32:46.231864 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:46.231867 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:46.231883 (+ 16us) server_negotiation.cc:407] Connection header received
0619 14:32:46.232062 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:46.232065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:46.232117 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:46.232239 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:46.232766 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:46.233426 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:46.234514 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:46.234891 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:46.234980 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:46.243439 (+ 8459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.2.1:39499: BlockingRecv error: recv got EOF from 127.2.2.1:39499 (error 108)
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":105,"threads_started":1}
Jun 19 14:32:46 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.2@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:32:46 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.2@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:32:46.434234 11237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:46.428413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46287 (local address 127.2.2.2:53133)
0619 14:32:46.428730 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:46.428747 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:46.428893 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:46.429191 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:46.429194 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:46.429414 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:46.429506 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:46.429510 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:46.430343 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:46.430349 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:46.430925 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:46.430933 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:46.431382 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:46.431400 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:46.431477 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:46.432048 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:46.432060 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:46.434024 (+ 1964us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.2.62:46287: Server oryx/127.2.2.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":80,"threads_started":1}
W20260619 14:32:46.434579 11236 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:32:46.428539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:53133 (local address 127.2.2.62:46287)
0619 14:32:46.428710 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:46.428715 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:46.428831 (+ 116us) server_negotiation.cc:407] Connection header received
0619 14:32:46.428995 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:46.428999 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:46.429053 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:46.429176 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:46.429632 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:46.430239 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:46.431046 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:46.431286 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:46.431395 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:46.434337 (+ 2942us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.2.2:53133: BlockingRecv error: recv got EOF from 127.2.2.2:53133 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Jun 19 14:32:46 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.3@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:32:46 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.3@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:32:46.619911 11238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:46.613871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46287 (local address 127.2.2.3:42825)
0619 14:32:46.614248 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:46.614262 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:46.614397 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:46.614691 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:46.614694 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:46.614866 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:46.614961 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:46.614966 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:46.615916 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:46.615920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:46.616766 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:46.616775 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:46.617196 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:46.617212 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:46.617271 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:46.617759 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:46.617771 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:46.619793 (+ 2022us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.2.62:46287: Server oryx/127.2.2.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":120,"threads_started":1}
W20260619 14:32:46.620092 11236 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:32:46.613951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.3:42825 (local address 127.2.2.62:46287)
0619 14:32:46.614103 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:46.614107 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:46.614346 (+ 239us) server_negotiation.cc:407] Connection header received
0619 14:32:46.614500 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:46.614502 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:46.614552 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:46.614621 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:46.615088 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:46.615788 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:46.616886 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:46.617103 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:46.617194 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:46.619951 (+ 2757us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.2.3:42825: BlockingRecv error: recv got EOF from 127.2.2.3:42825 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 19 14:32:47 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.1@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:32:47 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.1@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:32:47.249778 11239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:47.244317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46287 (local address 127.2.2.1:55361)
0619 14:32:47.244629 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:47.244642 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:47.244727 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:47.245027 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:47.245029 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:47.245206 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:47.245315 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:47.245320 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:47.246197 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:47.246199 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:47.246753 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:47.246761 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:47.247145 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:47.247159 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:47.247218 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:47.247817 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:47.247828 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:47.249643 (+ 1815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.2.62:46287: Server oryx/127.2.2.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":99,"threads_started":1}
W20260619 14:32:47.249967 11240 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:32:47.244386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:55361 (local address 127.2.2.62:46287)
0619 14:32:47.244695 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:47.244698 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:47.244713 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:32:47.244827 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:47.244830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:47.244902 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:47.244991 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:47.245422 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:47.246057 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:47.246848 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:47.247065 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:47.247167 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:47.249793 (+ 2626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.2.1:55361: BlockingRecv error: recv got EOF from 127.2.2.1:55361 (error 108)
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":80,"threads_started":1}
Jun 19 14:32:47 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.2@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:32:47 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.2@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:32:47.441059 11241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:47.435040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46287 (local address 127.2.2.2:33637)
0619 14:32:47.435363 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:47.435380 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:47.435496 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:47.435778 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:47.435781 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:47.435992 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:47.436079 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:47.436084 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:47.436842 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:47.436848 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:47.437504 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:47.437514 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:47.437969 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:47.437987 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:47.438075 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:47.438648 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:47.438659 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:47.440920 (+ 2261us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.2.62:46287: Server oryx/127.2.2.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":106,"threads_started":1}
W20260619 14:32:47.441140 11240 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:32:47.435238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:33637 (local address 127.2.2.62:46287)
0619 14:32:47.435381 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:47.435386 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:47.435454 (+ 68us) server_negotiation.cc:407] Connection header received
0619 14:32:47.435606 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:47.435609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:47.435659 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:47.435762 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:47.436194 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:47.436733 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:47.437619 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:47.437873 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:47.437980 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:47.441048 (+ 3068us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.2.2:33637: BlockingRecv error: recv got EOF from 127.2.2.2:33637 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 19 14:32:47 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.3@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:32:47 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.3@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:32:47.626617 11242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:47.620766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46287 (local address 127.2.2.3:42961)
0619 14:32:47.621085 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:47.621098 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:47.621203 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:47.621452 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:47.621454 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:47.621623 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:47.621718 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:47.621723 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:47.622541 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:47.622543 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:47.623097 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:47.623104 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:47.623559 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:47.623574 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:47.623633 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:47.624139 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:47.624151 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:47.626407 (+ 2256us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.2.62:46287: Server oryx/127.2.2.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":119,"threads_started":1}
W20260619 14:32:47.626753 11240 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:32:47.620823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.3:42961 (local address 127.2.2.62:46287)
0619 14:32:47.620974 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:47.620979 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:47.621224 (+ 245us) server_negotiation.cc:407] Connection header received
0619 14:32:47.621285 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:47.621287 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:47.621335 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:47.621420 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:47.621833 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:47.622445 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:47.623227 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:47.623457 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:47.623552 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:47.626604 (+ 3052us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.2.3:42961: BlockingRecv error: recv got EOF from 127.2.2.3:42961 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 19 14:32:48 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.1@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:32:48 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.1@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:32:48.256003 11243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:48.250648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46287 (local address 127.2.2.1:45965)
0619 14:32:48.250967 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:48.250982 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:48.251113 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:48.251417 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:48.251420 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:48.251605 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:48.251703 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:48.251709 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:48.252455 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:48.252458 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:48.253008 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:48.253014 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:48.253444 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:48.253459 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:48.253521 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:48.254018 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:48.254029 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:48.255889 (+ 1860us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.2.62:46287: Server oryx/127.2.2.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":80,"threads_started":1}
W20260619 14:32:48.256157 11244 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:32:48.250712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:45965 (local address 127.2.2.62:46287)
0619 14:32:48.250989 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:48.250994 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:48.251041 (+ 47us) server_negotiation.cc:407] Connection header received
0619 14:32:48.251206 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:48.251209 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:48.251290 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:48.251375 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:48.251802 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:48.252349 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:48.253114 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:48.253361 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:48.253439 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:48.256022 (+ 2583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.2.1:45965: BlockingRecv error: recv got EOF from 127.2.2.1:45965 (error 108)
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":62,"threads_started":1}
Jun 19 14:32:48 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.2@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:32:48 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.2@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:32:48.447189 11245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:48.441881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46287 (local address 127.2.2.2:37167)
0619 14:32:48.442212 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:48.442225 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:48.442320 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:48.442529 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:48.442531 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:48.442704 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:48.442811 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:48.442816 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:48.443568 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:48.443571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:48.444125 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:48.444133 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:48.444549 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:48.444582 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:48.444666 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:48.445193 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:48.445204 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:48.447064 (+ 1860us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.2.62:46287: Server oryx/127.2.2.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":94,"threads_started":1}
W20260619 14:32:48.447358 11244 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:32:48.441989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:37167 (local address 127.2.2.62:46287)
0619 14:32:48.442226 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:48.442230 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:48.442272 (+ 42us) server_negotiation.cc:407] Connection header received
0619 14:32:48.442386 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:48.442388 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:48.442429 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:48.442499 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:48.442910 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:48.443466 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:48.444253 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:48.444465 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:48.444543 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:48.447203 (+ 2660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.2.2:37167: BlockingRecv error: recv got EOF from 127.2.2.2:37167 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 19 14:32:48 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.3@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:32:48 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.3@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:32:48.633541 11246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:48.627724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46287 (local address 127.2.2.3:59109)
0619 14:32:48.628034 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:48.628048 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:48.628153 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:48.628435 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:48.628438 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:48.628658 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:48.628805 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:48.628813 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:48.629833 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:48.629837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:48.630533 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:48.630542 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:48.630957 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:48.630971 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:48.631030 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:48.631523 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:48.631543 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:48.633423 (+ 1880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.2.62:46287: Server oryx/127.2.2.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":132,"threads_started":1}
W20260619 14:32:48.633738 11244 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:32:48.627889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.3:59109 (local address 127.2.2.62:46287)
0619 14:32:48.628031 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:48.628035 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:48.628122 (+ 87us) server_negotiation.cc:407] Connection header received
0619 14:32:48.628240 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:48.628243 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:48.628291 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:48.628375 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:48.628955 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:48.629708 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:48.630652 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:48.630876 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:48.630956 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:48.633604 (+ 2648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.2.3:59109: BlockingRecv error: recv got EOF from 127.2.2.3:59109 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 19 14:32:49 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.1@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:32:49 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.1@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:32:49.263154 11247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:49.257004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46287 (local address 127.2.2.1:59201)
0619 14:32:49.257338 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:49.257356 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:49.257475 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:49.257729 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:49.257731 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:49.257958 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:49.258055 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:49.258061 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:49.259133 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:49.259137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:49.259912 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:49.259923 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:49.260385 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:49.260423 (+ 38us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:49.260517 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:49.261050 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:49.261062 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:49.263037 (+ 1975us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.2.62:46287: Server oryx/127.2.2.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":74,"threads_started":1}
W20260619 14:32:49.263352 11248 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:32:49.257203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:59201 (local address 127.2.2.62:46287)
0619 14:32:49.257469 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:49.257473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:49.257490 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:32:49.257554 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:49.257556 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:49.257599 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:49.257686 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:49.258217 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:49.258958 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:49.260046 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:49.260298 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:49.260379 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:49.263175 (+ 2796us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.2.1:59201: BlockingRecv error: recv got EOF from 127.2.2.1:59201 (error 108)
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":81,"threads_started":1}
Jun 19 14:32:49 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.2@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:32:49 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.2@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:32:49.453816 11249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:49.447962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46287 (local address 127.2.2.2:50043)
0619 14:32:49.448368 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:49.448381 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:49.448513 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:49.448776 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:49.448779 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:49.449001 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:49.449141 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:49.449147 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:49.449929 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:49.449934 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:49.450585 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:49.450593 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:49.451053 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:49.451072 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:49.451157 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:49.451728 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:49.451740 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:49.453646 (+ 1906us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.2.62:46287: Server oryx/127.2.2.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":95,"threads_started":1}
W20260619 14:32:49.453902 11248 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:32:49.448136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:50043 (local address 127.2.2.62:46287)
0619 14:32:49.448311 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:49.448316 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:49.448450 (+ 134us) server_negotiation.cc:407] Connection header received
0619 14:32:49.448612 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:49.448615 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:49.448669 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:49.448756 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:49.449268 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:49.449816 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:49.450699 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:49.450957 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:49.451067 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:49.453784 (+ 2717us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.2.2:50043: BlockingRecv error: recv got EOF from 127.2.2.2:50043 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 19 14:32:49 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.3@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:32:49 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.3@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:32:49.640779 11250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:49.634401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46287 (local address 127.2.2.3:43295)
0619 14:32:49.634729 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:49.634745 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:49.634856 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:49.635134 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:49.635136 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:49.635318 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:49.635414 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:49.635419 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:49.636300 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:49.636304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:49.637035 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:49.637046 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:49.637712 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:49.637748 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:49.637841 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:49.638539 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:49.638553 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:49.640644 (+ 2091us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.2.62:46287: Server oryx/127.2.2.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":89,"threads_started":1}
W20260619 14:32:49.641052 11248 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:32:49.634581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.3:43295 (local address 127.2.2.62:46287)
0619 14:32:49.634752 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:49.634756 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:49.634818 (+ 62us) server_negotiation.cc:407] Connection header received
0619 14:32:49.634940 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:49.634942 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:49.634990 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:49.635061 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:49.635510 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:49.636131 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:49.637225 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:49.637580 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:49.637695 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:49.640805 (+ 3110us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.2.3:43295: BlockingRecv error: recv got EOF from 127.2.2.3:43295 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 19 14:32:50 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.1@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:32:50 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.1@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:32:50.269747 11251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:50.263931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46287 (local address 127.2.2.1:54883)
0619 14:32:50.264238 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:50.264253 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:50.264344 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:50.264604 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:50.264606 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:50.264802 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:50.264903 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:50.264909 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:50.265727 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:50.265730 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:50.266440 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:50.266451 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:50.266910 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:50.266929 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:50.267016 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:50.267610 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:50.267630 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:50.269614 (+ 1984us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.2.62:46287: Server oryx/127.2.2.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":94,"threads_started":1}
W20260619 14:32:50.269958 11252 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:32:50.264128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:54883 (local address 127.2.2.62:46287)
0619 14:32:50.264346 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:50.264350 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:50.264363 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:32:50.264402 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:50.264404 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:50.264446 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:50.264525 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:50.265055 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:50.265601 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:50.266560 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:50.266794 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:50.266877 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:50.269772 (+ 2895us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.2.1:54883: BlockingRecv error: recv got EOF from 127.2.2.1:54883 (error 108)
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":66,"threads_started":1}
Jun 19 14:32:50 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.2@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:32:50 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.2@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:32:50.459964 11253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:50.454661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46287 (local address 127.2.2.2:35059)
0619 14:32:50.454922 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:50.454935 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:50.455031 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:50.455246 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:50.455248 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:50.455418 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:50.455512 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:50.455517 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:50.456394 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:50.456397 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:50.456960 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:50.456968 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:50.457376 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:50.457392 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:50.457455 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:50.457958 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:50.457971 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:50.459812 (+ 1841us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.2.62:46287: Server oryx/127.2.2.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":89,"threads_started":1}
W20260619 14:32:50.460132 11252 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:32:50.454720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:35059 (local address 127.2.2.62:46287)
0619 14:32:50.454904 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:50.454907 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:50.455004 (+ 97us) server_negotiation.cc:407] Connection header received
0619 14:32:50.455098 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:50.455100 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:50.455142 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:50.455211 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:50.455607 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:50.456287 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:50.457061 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:50.457286 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:50.457362 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:50.460011 (+ 2649us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.2.2:35059: BlockingRecv error: recv got EOF from 127.2.2.2:35059 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Jun 19 14:32:50 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.3@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:32:50 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.3@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:32:50.647657 11254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:50.641580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46287 (local address 127.2.2.3:52795)
0619 14:32:50.641920 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:50.641933 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:50.642079 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:50.642463 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:50.642466 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:50.642670 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:50.642803 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:50.642812 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:50.643570 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:50.643573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:50.644179 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:50.644188 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:50.644675 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:50.644699 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:50.644784 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:50.645369 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:50.645382 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:50.647523 (+ 2141us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.2.62:46287: Server oryx/127.2.2.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":98,"threads_started":1}
W20260619 14:32:50.647814 11252 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:32:50.641727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.3:52795 (local address 127.2.2.62:46287)
0619 14:32:50.641867 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:50.641876 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:50.642082 (+ 206us) server_negotiation.cc:407] Connection header received
0619 14:32:50.642257 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:50.642260 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:50.642342 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:50.642430 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:50.642934 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:50.643448 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:50.644293 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:50.644503 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:50.644579 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:50.647674 (+ 3095us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.2.3:52795: BlockingRecv error: recv got EOF from 127.2.2.3:52795 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 19 14:32:51 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.1@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:32:51 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.1@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:32:51.277671 11255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:51.270611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46287 (local address 127.2.2.1:57567)
0619 14:32:51.271283 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:51.271300 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:51.271413 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:51.271708 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:51.271710 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:51.271953 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:51.272090 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:51.272099 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:51.272874 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:51.272879 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:51.273959 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:51.273969 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:51.274479 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:51.274497 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:51.274583 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:51.275222 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:51.275238 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:51.277499 (+ 2261us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.2.62:46287: Server oryx/127.2.2.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":92,"threads_started":1}
W20260619 14:32:51.277787 11256 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:32:51.271008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:57567 (local address 127.2.2.62:46287)
0619 14:32:51.271274 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:51.271278 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:51.271359 (+ 81us) server_negotiation.cc:407] Connection header received
0619 14:32:51.271528 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:51.271531 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:51.271582 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:51.271690 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:51.272215 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:51.272755 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:51.274098 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:51.274382 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:51.274492 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:51.277633 (+ 3141us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.2.1:57567: BlockingRecv error: recv got EOF from 127.2.2.1:57567 (error 108)
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":79,"threads_started":1}
Jun 19 14:32:51 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.2@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:32:51 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.2@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:32:51.466450 11257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:51.460869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46287 (local address 127.2.2.2:54575)
0619 14:32:51.461145 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:51.461157 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:51.461290 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:51.461577 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:51.461580 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:51.461790 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:51.461876 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:51.461881 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:51.462682 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:51.462686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:51.463239 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:51.463246 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:51.463692 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:51.463709 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:51.463783 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:51.464343 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:51.464355 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:51.466267 (+ 1912us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.2.62:46287: Server oryx/127.2.2.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":78,"threads_started":1}
W20260619 14:32:51.466497 11256 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:32:51.460928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:54575 (local address 127.2.2.62:46287)
0619 14:32:51.461063 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:51.461068 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:51.461226 (+ 158us) server_negotiation.cc:407] Connection header received
0619 14:32:51.461412 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:51.461415 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:51.461460 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:51.461562 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:51.461989 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:51.462569 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:51.463372 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:51.463598 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:51.463706 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:51.466394 (+ 2688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.2.2:54575: BlockingRecv error: recv got EOF from 127.2.2.2:54575 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 19 14:32:51 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.3@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:32:51 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.3@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:32:51.653964 11258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:51.648603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46287 (local address 127.2.2.3:50705)
0619 14:32:51.648886 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:51.648899 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:51.649009 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:51.649269 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:51.649271 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:51.649451 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:51.649562 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:51.649568 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:51.650316 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:51.650319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:51.650916 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:51.650926 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:51.651349 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:51.651385 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:51.651458 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:51.651954 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:51.651966 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:51.653837 (+ 1871us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.2.62:46287: Server oryx/127.2.2.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":103,"threads_started":1}
W20260619 14:32:51.654102 11256 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:32:51.648601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.3:50705 (local address 127.2.2.62:46287)
0619 14:32:51.648707 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:51.648711 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:51.648964 (+ 253us) server_negotiation.cc:407] Connection header received
0619 14:32:51.649105 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:51.649108 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:51.649163 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:51.649230 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:51.649664 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:51.650200 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:51.651031 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:51.651250 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:51.651345 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:51.653972 (+ 2627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.2.3:50705: BlockingRecv error: recv got EOF from 127.2.2.3:50705 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Jun 19 14:32:52 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.1@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:32:52 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.1@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:32:52.285297 11259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:52.278845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46287 (local address 127.2.2.1:42495)
0619 14:32:52.279344 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:52.279359 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:52.279478 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:52.279798 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:52.279801 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:52.280035 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:52.280133 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:52.280148 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:52.281028 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:52.281035 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:52.281686 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:52.281695 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:52.282403 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:52.282424 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:52.282534 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:52.283137 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:52.283159 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:52.285180 (+ 2021us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.2.62:46287: Server oryx/127.2.2.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":88,"threads_started":1}
W20260619 14:32:52.285457 11260 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:32:52.279002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:42495 (local address 127.2.2.62:46287)
0619 14:32:52.279274 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:52.279278 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:52.279444 (+ 166us) server_negotiation.cc:407] Connection header received
0619 14:32:52.279569 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:52.279572 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:52.279665 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:52.279770 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:52.280276 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:52.280918 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:52.281819 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:52.282275 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:52.282406 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:52.285344 (+ 2938us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.2.1:42495: BlockingRecv error: recv got EOF from 127.2.2.1:42495 (error 108)
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":86,"threads_started":1}
Jun 19 14:32:52 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.2@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:32:52 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.2@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:32:52.473228 11261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:52.467410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46287 (local address 127.2.2.2:59927)
0619 14:32:52.467715 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:52.467729 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:52.467830 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:52.468144 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:52.468147 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:52.468336 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:52.468430 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:52.468435 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:52.469495 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:52.469501 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:52.470086 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:52.470095 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:52.470611 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:52.470634 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:52.470714 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:52.471282 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:52.471294 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:52.473112 (+ 1818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.2.62:46287: Server oryx/127.2.2.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":112,"threads_started":1}
W20260619 14:32:52.473433 11260 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:32:52.467561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:59927 (local address 127.2.2.62:46287)
0619 14:32:52.467724 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:52.467730 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:52.467808 (+ 78us) server_negotiation.cc:407] Connection header received
0619 14:32:52.467926 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:52.467932 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:52.467994 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:52.468134 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:52.468566 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:52.469334 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:52.470256 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:52.470488 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:52.470578 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:52.473261 (+ 2683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.2.2:59927: BlockingRecv error: recv got EOF from 127.2.2.2:59927 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 19 14:32:52 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.3@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:32:52 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.3@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:32:52.660830 11262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:52.654894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46287 (local address 127.2.2.3:37977)
0619 14:32:52.655229 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:52.655245 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:52.655391 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:52.655672 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:52.655675 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:52.655887 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:52.655977 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:52.655983 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:52.656801 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:52.656805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:52.657662 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:52.657690 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:52.658233 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:52.658254 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:52.658342 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:52.658821 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:52.658833 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:52.660693 (+ 1860us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.2.62:46287: Server oryx/127.2.2.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":99,"threads_started":1}
W20260619 14:32:52.661042 11260 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:32:52.655086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.3:37977 (local address 127.2.2.62:46287)
0619 14:32:52.655229 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:52.655235 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:52.655332 (+ 97us) server_negotiation.cc:407] Connection header received
0619 14:32:52.655494 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:52.655498 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:52.655554 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:52.655644 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:52.656104 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:52.656672 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:52.657807 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:52.658070 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:52.658248 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:52.660888 (+ 2640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.2.3:37977: BlockingRecv error: recv got EOF from 127.2.2.3:37977 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 19 14:32:53 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.1@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:32:53 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.1@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:32:53.291469 11263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:53.286330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46287 (local address 127.2.2.1:42523)
0619 14:32:53.286603 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:53.286617 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:53.286710 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:53.286922 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:53.286924 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:53.287098 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:53.287187 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:53.287191 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:53.287897 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:53.287899 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:53.288469 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:53.288478 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:53.288865 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:53.288879 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:53.288940 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:53.289505 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:53.289516 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:53.291352 (+ 1836us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.2.62:46287: Server oryx/127.2.2.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":100,"threads_started":1}
W20260619 14:32:53.291703 11264 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:32:53.286399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:42523 (local address 127.2.2.62:46287)
0619 14:32:53.286621 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:53.286625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:53.286665 (+ 40us) server_negotiation.cc:407] Connection header received
0619 14:32:53.286791 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:53.286793 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:53.286835 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:53.286914 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:53.287277 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:53.287799 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:53.288569 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:53.288784 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:53.288859 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:53.291539 (+ 2680us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.2.1:42523: BlockingRecv error: recv got EOF from 127.2.2.1:42523 (error 108)
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":84,"threads_started":1}
Jun 19 14:32:53 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.2@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:32:53 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.2@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:32:53.479851 11265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:53.474108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46287 (local address 127.2.2.2:40583)
0619 14:32:53.474450 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:53.474467 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:53.474630 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:53.474920 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:53.474923 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:53.475191 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:53.475286 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:53.475292 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:53.476170 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:53.476176 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:53.476738 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:53.476746 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:53.477194 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:53.477213 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:53.477299 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:53.477788 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:53.477800 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:53.479725 (+ 1925us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.2.62:46287: Server oryx/127.2.2.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":84,"threads_started":1}
W20260619 14:32:53.480136 11264 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:32:53.474265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:40583 (local address 127.2.2.62:46287)
0619 14:32:53.474445 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:53.474450 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:53.474564 (+ 114us) server_negotiation.cc:407] Connection header received
0619 14:32:53.474750 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:53.474754 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:53.474808 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:53.474899 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:53.475421 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:53.476053 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:53.476860 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:53.477088 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:53.477205 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:53.479880 (+ 2675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.2.2:40583: BlockingRecv error: recv got EOF from 127.2.2.2:40583 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jun 19 14:32:53 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.3@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:32:53 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.3@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:32:53.667734 11266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:53.661731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46287 (local address 127.2.2.3:47559)
0619 14:32:53.662023 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:53.662037 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:53.662262 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:53.662579 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:53.662583 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:53.662825 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:53.662927 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:53.662934 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:53.663780 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:53.663786 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:53.664391 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:53.664406 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:53.664917 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:53.664940 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:53.665033 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:53.665556 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:53.665569 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:53.667573 (+ 2004us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.2.62:46287: Server oryx/127.2.2.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":75,"threads_started":1}
W20260619 14:32:53.667852 11264 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:32:53.661789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.3:47559 (local address 127.2.2.62:46287)
0619 14:32:53.661945 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:53.661951 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:53.662172 (+ 221us) server_negotiation.cc:407] Connection header received
0619 14:32:53.662384 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:53.662387 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:53.662450 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:53.662552 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:53.663067 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:53.663648 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:53.664550 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:53.664807 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:53.664931 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:53.667740 (+ 2809us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.2.3:47559: BlockingRecv error: recv got EOF from 127.2.2.3:47559 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 19 14:32:54 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.1@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:32:54 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.1@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:32:54.298126 11267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:54.292298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46287 (local address 127.2.2.1:54717)
0619 14:32:54.292591 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:54.292606 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:54.292715 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:54.293010 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:54.293012 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:54.293215 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:54.293307 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:54.293312 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:54.294099 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:54.294101 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:54.294834 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:54.294843 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:54.295267 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:54.295283 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:54.295344 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:54.295824 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:54.295836 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:54.297959 (+ 2123us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.2.62:46287: Server oryx/127.2.2.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":86,"threads_started":1}
W20260619 14:32:54.298245 11268 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:32:54.292449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:54717 (local address 127.2.2.62:46287)
0619 14:32:54.292682 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:54.292685 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:54.292699 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:32:54.292838 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:54.292841 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:54.292890 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:54.292971 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:54.293454 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:54.293989 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:54.294948 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:54.295178 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:54.295258 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:54.298077 (+ 2819us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.2.1:54717: BlockingRecv error: recv got EOF from 127.2.2.1:54717 (error 108)
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":73,"threads_started":1}
Jun 19 14:32:54 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.2@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:32:54 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.2@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:32:54.486371 11269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:54.480632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46287 (local address 127.2.2.2:55837)
0619 14:32:54.480996 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:54.481018 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:54.481149 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:54.481448 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:54.481451 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:54.481664 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:54.481763 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:54.481768 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:54.482597 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:54.482602 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:54.483167 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:54.483175 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:54.483612 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:54.483630 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:54.483718 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:54.484236 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:54.484247 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:54.486254 (+ 2007us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.2.62:46287: Server oryx/127.2.2.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":123,"threads_started":1}
W20260619 14:32:54.486909 11268 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:32:54.480758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:55837 (local address 127.2.2.62:46287)
0619 14:32:54.480949 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:54.480953 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:54.481098 (+ 145us) server_negotiation.cc:407] Connection header received
0619 14:32:54.481280 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:54.481284 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:54.481336 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:54.481427 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:54.481887 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:54.482489 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:54.483285 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:54.483513 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:54.483622 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:54.486769 (+ 3147us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.2.2:55837: BlockingRecv error: recv got EOF from 127.2.2.2:55837 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Jun 19 14:32:54 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.3@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:32:54 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.3@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:32:54.673988 11270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:54.668559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46287 (local address 127.2.2.3:56943)
0619 14:32:54.668866 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:54.668880 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:54.668983 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:54.669239 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:54.669241 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:54.669420 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:54.669519 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:54.669525 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:54.670322 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:54.670324 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:54.670902 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:54.670912 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:54.671354 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:54.671373 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:54.671433 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:54.672013 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:54.672025 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:54.673852 (+ 1827us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.2.62:46287: Server oryx/127.2.2.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":129,"threads_started":1}
W20260619 14:32:54.674247 11268 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:32:54.668695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.3:56943 (local address 127.2.2.62:46287)
0619 14:32:54.668834 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:54.668837 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:54.668962 (+ 125us) server_negotiation.cc:407] Connection header received
0619 14:32:54.669070 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:54.669073 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:54.669122 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:54.669202 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:54.669619 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:54.670175 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:54.671028 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:54.671265 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:54.671348 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:54.674067 (+ 2719us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.2.3:56943: BlockingRecv error: recv got EOF from 127.2.2.3:56943 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 19 14:32:55 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.1@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:32:55 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.1@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:32:55.304873 11271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:55.299083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46287 (local address 127.2.2.1:35861)
0619 14:32:55.299402 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:55.299420 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:55.299512 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:55.299760 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:55.299762 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:55.299936 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:55.300027 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:55.300031 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:55.300806 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:55.300809 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:55.301362 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:55.301369 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:55.301818 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:55.301833 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:55.301892 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:55.302465 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:55.302477 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:55.304727 (+ 2250us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.2.62:46287: Server oryx/127.2.2.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":95,"threads_started":1}
W20260619 14:32:55.305061 11272 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:32:55.299161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:35861 (local address 127.2.2.62:46287)
0619 14:32:55.299452 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:55.299455 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:55.299470 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:32:55.299597 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:55.299599 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:55.299651 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:55.299731 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:55.300139 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:55.300699 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:55.301466 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:55.301697 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:55.301816 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:55.304877 (+ 3061us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.2.1:35861: BlockingRecv error: recv got EOF from 127.2.2.1:35861 (error 108)
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":118,"threads_started":1}
Jun 19 14:32:55 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.2@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:32:55 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.2@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:32:55.492538 11273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:55.487200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46287 (local address 127.2.2.2:44775)
0619 14:32:55.487487 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:55.487500 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:55.487586 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:55.487846 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:55.487848 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:55.488024 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:55.488115 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:55.488120 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:55.488830 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:55.488832 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:55.489369 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:55.489375 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:55.489748 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:55.489762 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:55.489832 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:55.490347 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:55.490358 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:55.492406 (+ 2048us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.2.62:46287: Server oryx/127.2.2.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":112,"threads_started":1}
W20260619 14:32:55.492621 11272 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:32:55.487399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:44775 (local address 127.2.2.62:46287)
0619 14:32:55.487535 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:55.487538 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:55.487552 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:32:55.487677 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:55.487679 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:55.487723 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:55.487800 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:55.488218 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:55.488735 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:55.489461 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:55.489671 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:55.489745 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:55.492530 (+ 2785us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.2.2:44775: BlockingRecv error: recv got EOF from 127.2.2.2:44775 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 19 14:32:55 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.3@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:32:55 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.3@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:32:55.680939 11274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:55.674886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46287 (local address 127.2.2.3:57789)
0619 14:32:55.675266 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:55.675282 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:55.675437 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:55.675739 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:55.675742 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:55.675962 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:55.676057 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:55.676063 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:55.676855 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:55.676859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:55.677434 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:55.677442 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:55.677893 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:55.677935 (+ 42us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:55.678024 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:55.678627 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:55.678639 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:55.680747 (+ 2108us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.2.62:46287: Server oryx/127.2.2.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":94,"threads_started":1}
W20260619 14:32:55.680994 11272 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:32:55.675037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.3:57789 (local address 127.2.2.62:46287)
0619 14:32:55.675229 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:55.675234 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:55.675381 (+ 147us) server_negotiation.cc:407] Connection header received
0619 14:32:55.675555 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:55.675558 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:55.675614 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:55.675710 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:55.676184 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:55.676730 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:55.677560 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:55.677791 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:55.677905 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:55.680878 (+ 2973us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.2.3:57789: BlockingRecv error: recv got EOF from 127.2.2.3:57789 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Jun 19 14:32:56 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.1@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:32:56 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.1@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:32:56.314554 11275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:56.305720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46287 (local address 127.2.2.1:43011)
0619 14:32:56.306012 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:56.306032 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:56.306178 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:56.306439 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:56.306441 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:56.306616 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:56.306707 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:56.306711 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:56.307518 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:56.307520 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:56.308064 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:56.308071 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:56.308512 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:56.308528 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:56.308589 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:56.312479 (+ 3890us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:56.312491 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:56.314372 (+ 1881us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.2.62:46287: Server oryx/127.2.2.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":74,"threads_started":1}
W20260619 14:32:56.314662 11276 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:32:56.305777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:43011 (local address 127.2.2.62:46287)
0619 14:32:56.306169 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:56.306175 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:56.306191 (+ 16us) server_negotiation.cc:407] Connection header received
0619 14:32:56.306280 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:56.306282 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:56.306326 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:56.306401 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:56.306837 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:56.307403 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:56.308192 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:56.308421 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:56.308500 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:56.314509 (+ 6009us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.2.1:43011: BlockingRecv error: recv got EOF from 127.2.2.1:43011 (error 108)
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":74,"threads_started":1}
Jun 19 14:32:56 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.2@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:32:56 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.2@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:32:56.498734 11277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:56.493490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46287 (local address 127.2.2.2:55357)
0619 14:32:56.493813 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:56.493827 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:56.493927 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:56.494228 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:56.494230 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:56.494420 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:56.494515 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:56.494520 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:56.495359 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:56.495361 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:56.495907 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:56.495913 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:56.496319 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:56.496333 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:56.496391 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:56.496866 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:56.496877 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:56.498611 (+ 1734us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.2.62:46287: Server oryx/127.2.2.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":117,"threads_started":1}
W20260619 14:32:56.498870 11276 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:32:56.493579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:55357 (local address 127.2.2.62:46287)
0619 14:32:56.493743 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:56.493749 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:56.493901 (+ 152us) server_negotiation.cc:407] Connection header received
0619 14:32:56.494010 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:56.494012 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:56.494056 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:56.494200 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:56.494618 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:56.495267 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:56.496002 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:56.496236 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:56.496312 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:56.498751 (+ 2439us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.2.2:55357: BlockingRecv error: recv got EOF from 127.2.2.2:55357 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 19 14:32:56 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.3@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:32:56 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.3@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:32:56.687729 11278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:56.682000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46287 (local address 127.2.2.3:51257)
0619 14:32:56.682334 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:56.682347 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:56.682450 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:56.682721 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:56.682723 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:56.682905 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:56.683001 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:56.683007 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:56.683807 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:56.683810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:56.684370 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:56.684378 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:56.684912 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:56.684946 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:56.685027 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:56.685563 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:56.685575 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:56.687605 (+ 2030us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.2.62:46287: Server oryx/127.2.2.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":85,"threads_started":1}
W20260619 14:32:56.687925 11276 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:32:56.682078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.3:51257 (local address 127.2.2.62:46287)
0619 14:32:56.682303 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:56.682307 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:56.682448 (+ 141us) server_negotiation.cc:407] Connection header received
0619 14:32:56.682551 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:56.682553 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:56.682602 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:56.682683 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:56.683168 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:56.683699 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:56.684515 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:56.684801 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:56.684906 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:56.687774 (+ 2868us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.2.3:51257: BlockingRecv error: recv got EOF from 127.2.2.3:51257 (error 108)
Metrics: {"server-negotiator.queue_time_us":123}
Jun 19 14:32:57 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.1@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:32:57 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.1@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:32:57.321501 11279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:57.315601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46287 (local address 127.2.2.1:46165)
0619 14:32:57.315885 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:57.315898 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:57.315983 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:57.316282 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:57.316285 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:57.316464 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:57.316561 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:57.316567 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:57.317469 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:57.317473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:57.318240 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:57.318252 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:57.318849 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:57.318866 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:57.318934 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:57.319500 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:57.319521 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:57.321368 (+ 1847us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.2.62:46287: Server oryx/127.2.2.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":98,"threads_started":1}
W20260619 14:32:57.321749 11280 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:32:57.315680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:46165 (local address 127.2.2.62:46287)
0619 14:32:57.315923 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:57.315926 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:57.315945 (+ 19us) server_negotiation.cc:407] Connection header received
0619 14:32:57.316073 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:57.316075 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:57.316117 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:57.316202 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:57.316677 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:57.317332 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:57.318362 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:57.318717 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:57.318825 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:57.321510 (+ 2685us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.2.1:46165: BlockingRecv error: recv got EOF from 127.2.2.1:46165 (error 108)
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":103,"threads_started":1}
Jun 19 14:32:57 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.2@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:32:57 dist-test-slave-70rm krb5kdc[10222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.2.2@KRBTEST.COM for oryx/127.2.2.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:32:57.505270 11281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:57.499765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46287 (local address 127.2.2.2:40397)
0619 14:32:57.500034 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:57.500048 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:57.500138 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:57.500455 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:57.500457 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:57.500643 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:57.500734 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:57.500739 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:57.501459 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:57.501462 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:57.502029 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:57.502037 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:57.502527 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:57.502545 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:57.502608 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:57.503157 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:32:57.503169 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:57.505152 (+ 1983us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.2.62:46287: Server oryx/127.2.2.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":67,"threads_started":1}
W20260619 14:32:57.505437 11280 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:32:57.499909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:40397 (local address 127.2.2.62:46287)
0619 14:32:57.500077 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:57.500080 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:57.500096 (+ 16us) server_negotiation.cc:407] Connection header received
0619 14:32:57.500282 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:57.500285 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:57.500335 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:57.500411 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:57.500835 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:57.501358 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:57.502207 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:57.502434 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:57.502522 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:57.505330 (+ 2808us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.2.2:40397: BlockingRecv error: recv got EOF from 127.2.2.2:40397 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 14:32:57.576453 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 10805
I20260619 14:32:57.582581 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 10957
I20260619 14:32:57.588878 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 11088
I20260619 14:32:57.594842 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 10735
2026-06-19T14:32:57Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18045 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-19T14:32:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:32:57Z Disabled control of system clock
I20260619 14:32:57.624881 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:33839
--webserver_interface=127.2.2.62
--webserver_port=0
--builtin_ntp_servers=127.2.2.20:44703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.2.62:33839
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 14:32:57.740237 11285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:57.740487 11285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:57.740521 11285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:57.744616 11285 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:32:57.744678 11285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:57.744694 11285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:57.744711 11285 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:32:57.744726 11285 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:32:57.749498 11285 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:44703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.2.62:33839
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:33839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.2.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:32:57.750748 11285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:57.752219 11285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:32:57.760968 11285 server_base.cc:1034] running on GCE node
W20260619 14:32:57.760918 11291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:57.760918 11290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:57.760936 11293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:57.761714 11285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:57.762953 11285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:57.764142 11285 hybrid_clock.cc:648] HybridClock initialized: now 1781879577764110 us; error 49 us; skew 500 ppm
I20260619 14:32:57.766041 11285 webserver.cc:460] Webserver started at http://127.2.2.62:39815/ using document root <none> and password file <none>
I20260619 14:32:57.766652 11285 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:57.766752 11285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:57.766996 11285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:32:57.769050 11285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/master-0/data/instance:
uuid: "e70a2c1faac64d8991d03f63553ae714"
format_stamp: "Formatted at 2026-06-19 14:32:57 on dist-test-slave-70rm"
server_key: "891b335bc386e9e663f630aef75f3b53"
server_key_iv: "3d5d54cd8a9e3107ec7fc460fdada455"
server_key_version: "encryptionkey@0"
I20260619 14:32:57.769610 11285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/master-0/wal/instance:
uuid: "e70a2c1faac64d8991d03f63553ae714"
format_stamp: "Formatted at 2026-06-19 14:32:57 on dist-test-slave-70rm"
server_key: "891b335bc386e9e663f630aef75f3b53"
server_key_iv: "3d5d54cd8a9e3107ec7fc460fdada455"
server_key_version: "encryptionkey@0"
I20260619 14:32:57.773355 11285 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 14:32:57.775867 11299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:57.777145 11285 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:32:57.777287 11285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/master-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/master-0/wal
uuid: "e70a2c1faac64d8991d03f63553ae714"
format_stamp: "Formatted at 2026-06-19 14:32:57 on dist-test-slave-70rm"
server_key: "891b335bc386e9e663f630aef75f3b53"
server_key_iv: "3d5d54cd8a9e3107ec7fc460fdada455"
server_key_version: "encryptionkey@0"
I20260619 14:32:57.777410 11285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:57.800226 11285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:57.800977 11285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:32:57.801183 11285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:57.808444 11285 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.62:33839
I20260619 14:32:57.808456 11351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.62:33839 every 8 connection(s)
I20260619 14:32:57.809391 11285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
I20260619 14:32:57.811082 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 11285
I20260619 14:32:57.811206 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/master-0/wal/instance
I20260619 14:32:57.811499 2056 external_mini_cluster.cc:1428] Setting key a3311971e9acc3cc49dc1a84dd751179
I20260619 14:32:57.813301 11352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:57.820327 11352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e70a2c1faac64d8991d03f63553ae714: Bootstrap starting.
I20260619 14:32:57.821163 11355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:57.812840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53086 (local address 127.2.2.62:33839)
0619 14:32:57.813498 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:57.813515 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:57.813552 (+ 37us) server_negotiation.cc:407] Connection header received
0619 14:32:57.814274 (+ 722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:57.814293 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:57.814695 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:57.814993 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:57.815382 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:57.816384 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:57.817742 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:57.818289 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:57.818405 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:57.819376 (+ 971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:57.819381 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:57.819394 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:32:57.819799 (+ 405us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:57.819855 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:32:57.819875 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:57.819951 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:57.820119 (+ 168us) server_negotiation.cc:299] Negotiation successful
0619 14:32:57.820457 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":142,"threads_started":1}
I20260619 14:32:57.822741 11352 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e70a2c1faac64d8991d03f63553ae714: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:57.823625 11352 log.cc:824] T 00000000000000000000000000000000 P e70a2c1faac64d8991d03f63553ae714: Log is configured to *not* fsync() on all Append() calls
I20260619 14:32:57.825659 11352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e70a2c1faac64d8991d03f63553ae714: No bootstrap required, opened a new log
I20260619 14:32:57.829032 11352 raft_consensus.cc:348] T 00000000000000000000000000000000 P e70a2c1faac64d8991d03f63553ae714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e70a2c1faac64d8991d03f63553ae714" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 33839 } }
I20260619 14:32:57.829293 11352 raft_consensus.cc:374] T 00000000000000000000000000000000 P e70a2c1faac64d8991d03f63553ae714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:57.829331 11352 raft_consensus.cc:729] T 00000000000000000000000000000000 P e70a2c1faac64d8991d03f63553ae714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e70a2c1faac64d8991d03f63553ae714, State: Initialized, Role: FOLLOWER
I20260619 14:32:57.829746 11352 consensus_queue.cc:260] T 00000000000000000000000000000000 P e70a2c1faac64d8991d03f63553ae714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e70a2c1faac64d8991d03f63553ae714" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 33839 } }
I20260619 14:32:57.829864 11352 raft_consensus.cc:388] T 00000000000000000000000000000000 P e70a2c1faac64d8991d03f63553ae714 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:32:57.829910 11352 raft_consensus.cc:482] T 00000000000000000000000000000000 P e70a2c1faac64d8991d03f63553ae714 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:32:57.829980 11352 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e70a2c1faac64d8991d03f63553ae714 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:57.831106 11352 raft_consensus.cc:504] T 00000000000000000000000000000000 P e70a2c1faac64d8991d03f63553ae714 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e70a2c1faac64d8991d03f63553ae714" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 33839 } }
I20260619 14:32:57.831509 11352 leader_election.cc:302] T 00000000000000000000000000000000 P e70a2c1faac64d8991d03f63553ae714 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e70a2c1faac64d8991d03f63553ae714; no voters:
I20260619 14:32:57.831849 11352 leader_election.cc:288] T 00000000000000000000000000000000 P e70a2c1faac64d8991d03f63553ae714 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:32:57.831988 11357 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e70a2c1faac64d8991d03f63553ae714 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:32:57.832252 11357 raft_consensus.cc:686] T 00000000000000000000000000000000 P e70a2c1faac64d8991d03f63553ae714 [term 1 LEADER]: Becoming Leader. State: Replica: e70a2c1faac64d8991d03f63553ae714, State: Running, Role: LEADER
I20260619 14:32:57.832602 11357 consensus_queue.cc:237] T 00000000000000000000000000000000 P e70a2c1faac64d8991d03f63553ae714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e70a2c1faac64d8991d03f63553ae714" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 33839 } }
I20260619 14:32:57.833099 11352 sys_catalog.cc:564] T 00000000000000000000000000000000 P e70a2c1faac64d8991d03f63553ae714 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:32:57.834252 11358 sys_catalog.cc:455] T 00000000000000000000000000000000 P e70a2c1faac64d8991d03f63553ae714 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e70a2c1faac64d8991d03f63553ae714" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e70a2c1faac64d8991d03f63553ae714" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 33839 } } }
I20260619 14:32:57.834388 11358 sys_catalog.cc:458] T 00000000000000000000000000000000 P e70a2c1faac64d8991d03f63553ae714 [sys.catalog]: This master's current role is: LEADER
I20260619 14:32:57.834733 11359 sys_catalog.cc:455] T 00000000000000000000000000000000 P e70a2c1faac64d8991d03f63553ae714 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e70a2c1faac64d8991d03f63553ae714. Latest consensus state: current_term: 1 leader_uuid: "e70a2c1faac64d8991d03f63553ae714" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e70a2c1faac64d8991d03f63553ae714" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 33839 } } }
I20260619 14:32:57.834794 11366 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:32:57.834918 11359 sys_catalog.cc:458] T 00000000000000000000000000000000 P e70a2c1faac64d8991d03f63553ae714 [sys.catalog]: This master's current role is: LEADER
I20260619 14:32:57.838004 11366 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:32:57.844193 11366 catalog_manager.cc:1347] Generated new cluster ID: 6a1b12a33eab4446b6216194068c9988
I20260619 14:32:57.844273 11366 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:32:57.855837 11366 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:32:57.856846 11366 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:32:57.864516 11366 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e70a2c1faac64d8991d03f63553ae714: Generated new TSK 0
I20260619 14:32:57.865264 11366 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:32:57.877630 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.1:0
--local_ip_for_outbound_sockets=127.2.2.1
--webserver_interface=127.2.2.1
--webserver_port=0
--tserver_master_addrs=127.2.2.62:33839
--builtin_ntp_servers=127.2.2.20:44703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260619 14:32:57.991288 11376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:57.991580 11376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:57.991618 11376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:57.995698 11376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:57.995757 11376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:57.995810 11376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.1
I20260619 14:32:58.000507 11376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:44703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.2.1
--webserver_port=0
--tserver_master_addrs=127.2.2.62:33839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.1
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:32:58.001617 11376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:58.003006 11376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:32:58.012518 11381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:58.012540 11382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:58.012701 11376 server_base.cc:1034] running on GCE node
W20260619 14:32:58.012524 11384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:58.013283 11376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:58.013916 11376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:58.015092 11376 hybrid_clock.cc:648] HybridClock initialized: now 1781879578015078 us; error 57 us; skew 500 ppm
I20260619 14:32:58.017120 11376 webserver.cc:460] Webserver started at http://127.2.2.1:43247/ using document root <none> and password file <none>
I20260619 14:32:58.017728 11376 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:58.017794 11376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:58.018036 11376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:32:58.020143 11376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-0/data/instance:
uuid: "d46a0ac983c44a77b0325a18f64ada56"
format_stamp: "Formatted at 2026-06-19 14:32:58 on dist-test-slave-70rm"
server_key: "1fd00bb1be0bc9df1c6728a96669dbe0"
server_key_iv: "8e249dd99e3b6fe2eac5bdd1c303558c"
server_key_version: "encryptionkey@0"
I20260619 14:32:58.020727 11376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-0/wal/instance:
uuid: "d46a0ac983c44a77b0325a18f64ada56"
format_stamp: "Formatted at 2026-06-19 14:32:58 on dist-test-slave-70rm"
server_key: "1fd00bb1be0bc9df1c6728a96669dbe0"
server_key_iv: "8e249dd99e3b6fe2eac5bdd1c303558c"
server_key_version: "encryptionkey@0"
I20260619 14:32:58.024637 11376 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:32:58.027345 11390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:58.028739 11376 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 14:32:58.028882 11376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-0/wal
uuid: "d46a0ac983c44a77b0325a18f64ada56"
format_stamp: "Formatted at 2026-06-19 14:32:58 on dist-test-slave-70rm"
server_key: "1fd00bb1be0bc9df1c6728a96669dbe0"
server_key_iv: "8e249dd99e3b6fe2eac5bdd1c303558c"
server_key_version: "encryptionkey@0"
I20260619 14:32:58.029008 11376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:58.044900 11376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:58.045653 11376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:32:58.045866 11376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:58.046514 11376 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:32:58.047477 11376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:32:58.047521 11376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:58.047593 11376 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:32:58.047624 11376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:58.057040 11376 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.1:41517
I20260619 14:32:58.057063 11503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.1:41517 every 8 connection(s)
I20260619 14:32:58.058063 11376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
I20260619 14:32:58.064589 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 11376
I20260619 14:32:58.064713 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-0/wal/instance
I20260619 14:32:58.065033 2056 external_mini_cluster.cc:1428] Setting key 35fa219b9421e3f5364d02834c43f1ca
I20260619 14:32:58.067339 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.2:0
--local_ip_for_outbound_sockets=127.2.2.2
--webserver_interface=127.2.2.2
--webserver_port=0
--tserver_master_addrs=127.2.2.62:33839
--builtin_ntp_servers=127.2.2.20:44703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 14:32:58.069063 11355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:58.060088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:47535 (local address 127.2.2.62:33839)
0619 14:32:58.060270 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:58.060274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:58.061423 (+ 1149us) server_negotiation.cc:407] Connection header received
0619 14:32:58.062863 (+ 1440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:58.062874 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:58.062924 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:58.063014 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:58.064067 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:58.064855 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:58.066677 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:58.066977 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:58.067105 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:58.068457 (+ 1352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:58.068464 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:58.068470 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:32:58.068488 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:58.068517 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:32:58.068531 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:58.068620 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:58.068872 (+ 252us) server_negotiation.cc:299] Negotiation successful
0619 14:32:58.068938 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 14:32:58.070528 11506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:58.060513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:33839 (local address 127.2.2.1:47535)
0619 14:32:58.061264 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:58.061310 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:58.062628 (+ 1318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:58.063240 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:58.063248 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:58.063366 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:58.063919 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:58.063933 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:58.065015 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:58.065018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:58.066521 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:58.066535 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:58.067119 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:58.067178 (+ 59us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:58.067340 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:58.068150 (+ 810us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:32:58.068151 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:58.068202 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:32:58.068281 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:58.068661 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:58.068675 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:58.068684 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 14:32:58.068869 (+ 185us) client_negotiation.cc:240] Negotiation successful
0619 14:32:58.069233 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":121,"threads_started":1}
I20260619 14:32:58.071938 11504 heartbeater.cc:347] Connected to a master server at 127.2.2.62:33839
I20260619 14:32:58.072250 11504 heartbeater.cc:464] Registering TS with master...
I20260619 14:32:58.073019 11504 heartbeater.cc:511] Master 127.2.2.62:33839 requested a full tablet report, sending...
I20260619 14:32:58.075186 11316 ts_manager.cc:194] Registered new tserver with Master: d46a0ac983c44a77b0325a18f64ada56 (127.2.2.1:41517)
I20260619 14:32:58.076560 11316 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.2.1:47535
W20260619 14:32:58.182329 11507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:58.182621 11507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:58.182658 11507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:58.186779 11507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:58.186847 11507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:58.186904 11507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.2
I20260619 14:32:58.191696 11507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:44703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.2.2
--webserver_port=0
--tserver_master_addrs=127.2.2.62:33839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.2
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:32:58.192847 11507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:58.194278 11507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:32:58.204124 11515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:58.204056 11512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:58.204221 11507 server_base.cc:1034] running on GCE node
W20260619 14:32:58.204051 11513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:58.204830 11507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:58.205510 11507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:58.206669 11507 hybrid_clock.cc:648] HybridClock initialized: now 1781879578206652 us; error 34 us; skew 500 ppm
I20260619 14:32:58.208699 11507 webserver.cc:460] Webserver started at http://127.2.2.2:46687/ using document root <none> and password file <none>
I20260619 14:32:58.209288 11507 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:58.209381 11507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:58.209607 11507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:32:58.211779 11507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-1/data/instance:
uuid: "cc98dcecf47642da893816266557588e"
format_stamp: "Formatted at 2026-06-19 14:32:58 on dist-test-slave-70rm"
server_key: "85301b8e7dcaf8a001d0a3a94ca5e596"
server_key_iv: "f4a210cc678f09111439502e62771dfa"
server_key_version: "encryptionkey@0"
I20260619 14:32:58.212365 11507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-1/wal/instance:
uuid: "cc98dcecf47642da893816266557588e"
format_stamp: "Formatted at 2026-06-19 14:32:58 on dist-test-slave-70rm"
server_key: "85301b8e7dcaf8a001d0a3a94ca5e596"
server_key_iv: "f4a210cc678f09111439502e62771dfa"
server_key_version: "encryptionkey@0"
I20260619 14:32:58.216302 11507 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:32:58.219019 11521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:58.220300 11507 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:32:58.220434 11507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-1/wal
uuid: "cc98dcecf47642da893816266557588e"
format_stamp: "Formatted at 2026-06-19 14:32:58 on dist-test-slave-70rm"
server_key: "85301b8e7dcaf8a001d0a3a94ca5e596"
server_key_iv: "f4a210cc678f09111439502e62771dfa"
server_key_version: "encryptionkey@0"
I20260619 14:32:58.220554 11507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:58.240454 11507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:58.241218 11507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:32:58.241425 11507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:58.242030 11507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:32:58.242995 11507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:32:58.243067 11507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:58.243132 11507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:32:58.243180 11507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:58.253751 11507 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.2:38493
I20260619 14:32:58.253777 11634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.2:38493 every 8 connection(s)
I20260619 14:32:58.254813 11507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
I20260619 14:32:58.264552 11355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:58.256784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:50541 (local address 127.2.2.62:33839)
0619 14:32:58.256935 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:58.256938 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:58.258112 (+ 1174us) server_negotiation.cc:407] Connection header received
0619 14:32:58.259625 (+ 1513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:58.259629 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:58.259699 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:58.259808 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:58.260757 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:58.261282 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:58.262473 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:58.262705 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:58.262821 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:58.264014 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:58.264017 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:58.264022 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:32:58.264039 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:58.264061 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:32:58.264069 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:58.264150 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:58.264325 (+ 175us) server_negotiation.cc:299] Negotiation successful
0619 14:32:58.264397 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 14:32:58.265009 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 11507
I20260619 14:32:58.265093 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-1/wal/instance
I20260619 14:32:58.265386 2056 external_mini_cluster.cc:1428] Setting key af1a31a457e0d28a2bfa8983668fcfbc
I20260619 14:32:58.265568 11637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:58.257221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:33839 (local address 127.2.2.2:50541)
0619 14:32:58.257933 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:58.257976 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:58.259389 (+ 1413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:58.259995 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:58.260002 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:58.260091 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:58.260590 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:58.260602 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:58.261417 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:58.261425 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:58.262347 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:58.262355 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:58.262809 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:58.262865 (+ 56us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:58.263032 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:58.263712 (+ 680us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:32:58.263714 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:58.263765 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:32:58.263832 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:58.264173 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:58.264183 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:58.264193 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 14:32:58.264325 (+ 132us) client_negotiation.cc:240] Negotiation successful
0619 14:32:58.264627 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":125,"threads_started":1}
I20260619 14:32:58.266870 11635 heartbeater.cc:347] Connected to a master server at 127.2.2.62:33839
I20260619 14:32:58.267124 11635 heartbeater.cc:464] Registering TS with master...
I20260619 14:32:58.267642 11635 heartbeater.cc:511] Master 127.2.2.62:33839 requested a full tablet report, sending...
I20260619 14:32:58.267652 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.3:0
--local_ip_for_outbound_sockets=127.2.2.3
--webserver_interface=127.2.2.3
--webserver_port=0
--tserver_master_addrs=127.2.2.62:33839
--builtin_ntp_servers=127.2.2.20:44703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 14:32:58.268703 11315 ts_manager.cc:194] Registered new tserver with Master: cc98dcecf47642da893816266557588e (127.2.2.2:38493)
I20260619 14:32:58.269544 11315 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.2.2:50541
W20260619 14:32:58.381991 11638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:58.382354 11638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:58.382395 11638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:58.386570 11638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:58.386644 11638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:58.386701 11638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.3
I20260619 14:32:58.391522 11638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:44703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.2.3
--webserver_port=0
--tserver_master_addrs=127.2.2.62:33839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.3
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:32:58.392625 11638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:58.394033 11638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:32:58.404745 11646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:58.404767 11644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:58.404747 11643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:58.404989 11638 server_base.cc:1034] running on GCE node
I20260619 14:32:58.405541 11638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:58.406255 11638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:58.407425 11638 hybrid_clock.cc:648] HybridClock initialized: now 1781879578407406 us; error 35 us; skew 500 ppm
I20260619 14:32:58.409422 11638 webserver.cc:460] Webserver started at http://127.2.2.3:46733/ using document root <none> and password file <none>
I20260619 14:32:58.410002 11638 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:58.410065 11638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:58.410354 11638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:32:58.412446 11638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-2/data/instance:
uuid: "64daedf1459c471c876ed1e85816850f"
format_stamp: "Formatted at 2026-06-19 14:32:58 on dist-test-slave-70rm"
server_key: "b937585618bb2ae16e5f57db2024e493"
server_key_iv: "3cd5a288597651dcfde61f51b43addfc"
server_key_version: "encryptionkey@0"
I20260619 14:32:58.412999 11638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-2/wal/instance:
uuid: "64daedf1459c471c876ed1e85816850f"
format_stamp: "Formatted at 2026-06-19 14:32:58 on dist-test-slave-70rm"
server_key: "b937585618bb2ae16e5f57db2024e493"
server_key_iv: "3cd5a288597651dcfde61f51b43addfc"
server_key_version: "encryptionkey@0"
I20260619 14:32:58.416716 11638 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260619 14:32:58.419374 11652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:58.420677 11638 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:32:58.420811 11638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-2/wal
uuid: "64daedf1459c471c876ed1e85816850f"
format_stamp: "Formatted at 2026-06-19 14:32:58 on dist-test-slave-70rm"
server_key: "b937585618bb2ae16e5f57db2024e493"
server_key_iv: "3cd5a288597651dcfde61f51b43addfc"
server_key_version: "encryptionkey@0"
I20260619 14:32:58.420938 11638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:58.434327 11638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:58.435115 11638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:32:58.435321 11638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:58.435964 11638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:32:58.436933 11638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:32:58.437006 11638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:58.437078 11638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:32:58.437121 11638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:58.446345 11638 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.3:33643
I20260619 14:32:58.446357 11765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.3:33643 every 8 connection(s)
I20260619 14:32:58.447294 11638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
I20260619 14:32:58.454833 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 11638
I20260619 14:32:58.454938 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-2/wal/instance
I20260619 14:32:58.455214 2056 external_mini_cluster.cc:1428] Setting key 931d727c329100cb44757df10a0eceb9
I20260619 14:32:58.457719 11355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:58.449212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.3:35433 (local address 127.2.2.62:33839)
0619 14:32:58.449363 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:58.449367 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:58.450528 (+ 1161us) server_negotiation.cc:407] Connection header received
0619 14:32:58.452076 (+ 1548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:58.452079 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:58.452160 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:58.452243 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:58.453189 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:58.453707 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:58.455342 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:32:58.455631 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:32:58.455742 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:58.457107 (+ 1365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:58.457111 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:58.457114 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:32:58.457164 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:58.457188 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:32:58.457199 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:58.457280 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:58.457492 (+ 212us) server_negotiation.cc:299] Negotiation successful
0619 14:32:58.457574 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 14:32:58.458675 11768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:58.449622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:33839 (local address 127.2.2.3:35433)
0619 14:32:58.450375 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:58.450418 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:58.451695 (+ 1277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:58.452434 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:58.452441 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:58.452540 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:58.453030 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:58.453043 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:58.453909 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:58.453912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:58.455193 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:32:58.455205 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:32:58.455745 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:32:58.455799 (+ 54us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:32:58.455987 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:32:58.456826 (+ 839us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:32:58.456830 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:58.456879 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:32:58.456947 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:58.457294 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:58.457316 (+ 22us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:58.457320 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 14:32:58.457457 (+ 137us) client_negotiation.cc:240] Negotiation successful
0619 14:32:58.457756 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":113,"threads_started":1}
I20260619 14:32:58.459782 11766 heartbeater.cc:347] Connected to a master server at 127.2.2.62:33839
I20260619 14:32:58.460047 11766 heartbeater.cc:464] Registering TS with master...
I20260619 14:32:58.460613 11766 heartbeater.cc:511] Master 127.2.2.62:33839 requested a full tablet report, sending...
I20260619 14:32:58.461633 11315 ts_manager.cc:194] Registered new tserver with Master: 64daedf1459c471c876ed1e85816850f (127.2.2.3:33643)
I20260619 14:32:58.462267 11315 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.2.3:35433
I20260619 14:32:58.469499 2056 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 14:32:58.472453 11355 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:32:58.471463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53092 (local address 127.2.2.62:33839)
0619 14:32:58.471626 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:58.471632 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:58.471791 (+ 159us) server_negotiation.cc:407] Connection header received
0619 14:32:58.471948 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:58.471951 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:58.471996 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:58.472088 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:58.472323 (+ 235us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53092: BlockingRecv error: recv got EOF from 127.0.0.1:53092 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 14:32:58.472605 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 11376
I20260619 14:32:58.478521 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 11507
I20260619 14:32:58.483996 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 11638
I20260619 14:32:58.489179 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 11285
2026-06-19T14:32:58Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (894 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-19T14:32:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:32:58Z Disabled control of system clock
I20260619 14:32:58.518776 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:37291
--webserver_interface=127.2.2.62
--webserver_port=0
--builtin_ntp_servers=127.2.2.20:38311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.2.62:37291
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 14:32:58.629778 11777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:58.630071 11777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:58.630110 11777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:58.634763 11777 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:32:58.634850 11777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:58.634878 11777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:58.634910 11777 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:32:58.634945 11777 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:32:58.640722 11777 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:38311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.2.62:37291
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:37291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.2.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:32:58.641923 11777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:58.643399 11777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:32:58.652132 11785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:58.652125 11783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:58.652230 11777 server_base.cc:1034] running on GCE node
W20260619 14:32:58.652170 11782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:58.653043 11777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:58.654367 11777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:58.655536 11777 hybrid_clock.cc:648] HybridClock initialized: now 1781879578655525 us; error 40 us; skew 500 ppm
I20260619 14:32:58.657570 11777 webserver.cc:460] Webserver started at http://127.2.2.62:44955/ using document root <none> and password file <none>
I20260619 14:32:58.658248 11777 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:58.658321 11777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:58.658547 11777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:32:58.660699 11777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/master-0/data/instance:
uuid: "1171af5a19224b379c9e709ad88d37a3"
format_stamp: "Formatted at 2026-06-19 14:32:58 on dist-test-slave-70rm"
server_key: "f5bb648984343120e9398f391b05603d"
server_key_iv: "5c5abc197aee6cbd5e922f866db040bc"
server_key_version: "encryptionkey@0"
I20260619 14:32:58.661332 11777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/master-0/wal/instance:
uuid: "1171af5a19224b379c9e709ad88d37a3"
format_stamp: "Formatted at 2026-06-19 14:32:58 on dist-test-slave-70rm"
server_key: "f5bb648984343120e9398f391b05603d"
server_key_iv: "5c5abc197aee6cbd5e922f866db040bc"
server_key_version: "encryptionkey@0"
I20260619 14:32:58.665333 11777 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 14:32:58.668071 11791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:58.669430 11777 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:32:58.669585 11777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/master-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/master-0/wal
uuid: "1171af5a19224b379c9e709ad88d37a3"
format_stamp: "Formatted at 2026-06-19 14:32:58 on dist-test-slave-70rm"
server_key: "f5bb648984343120e9398f391b05603d"
server_key_iv: "5c5abc197aee6cbd5e922f866db040bc"
server_key_version: "encryptionkey@0"
I20260619 14:32:58.669716 11777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:58.674557 11777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:58.675210 11777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:32:58.675417 11777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:58.682376 11777 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.62:37291
I20260619 14:32:58.682407 11843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.62:37291 every 8 connection(s)
I20260619 14:32:58.683357 11777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
I20260619 14:32:58.684471 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 11777
I20260619 14:32:58.684602 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/master-0/wal/instance
I20260619 14:32:58.684923 2056 external_mini_cluster.cc:1428] Setting key df914ea3ae1e1b0ac313a513312f4a17
I20260619 14:32:58.686861 11844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:32:58.691391 11847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:58.686441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54246 (local address 127.2.2.62:37291)
0619 14:32:58.687759 (+ 1318us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:58.687774 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:58.687811 (+ 37us) server_negotiation.cc:407] Connection header received
0619 14:32:58.688456 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:58.688472 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:58.688834 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:58.689186 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:58.690058 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:58.690063 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:58.690076 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:32:58.690086 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:58.690184 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:32:58.690214 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:58.690369 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:58.690441 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 14:32:58.690687 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":940,"thread_start_us":780,"threads_started":1}
I20260619 14:32:58.693774 11844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1171af5a19224b379c9e709ad88d37a3: Bootstrap starting.
I20260619 14:32:58.696089 11844 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1171af5a19224b379c9e709ad88d37a3: Neither blocks nor log segments found. Creating new log.
I20260619 14:32:58.696870 11844 log.cc:824] T 00000000000000000000000000000000 P 1171af5a19224b379c9e709ad88d37a3: Log is configured to *not* fsync() on all Append() calls
I20260619 14:32:58.698993 11844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1171af5a19224b379c9e709ad88d37a3: No bootstrap required, opened a new log
I20260619 14:32:58.703045 11844 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1171af5a19224b379c9e709ad88d37a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1171af5a19224b379c9e709ad88d37a3" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 37291 } }
I20260619 14:32:58.703336 11844 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1171af5a19224b379c9e709ad88d37a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:32:58.703413 11844 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1171af5a19224b379c9e709ad88d37a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1171af5a19224b379c9e709ad88d37a3, State: Initialized, Role: FOLLOWER
I20260619 14:32:58.703954 11844 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1171af5a19224b379c9e709ad88d37a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1171af5a19224b379c9e709ad88d37a3" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 37291 } }
I20260619 14:32:58.704136 11844 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1171af5a19224b379c9e709ad88d37a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:32:58.704195 11844 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1171af5a19224b379c9e709ad88d37a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:32:58.704277 11844 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1171af5a19224b379c9e709ad88d37a3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:32:58.705211 11844 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1171af5a19224b379c9e709ad88d37a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1171af5a19224b379c9e709ad88d37a3" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 37291 } }
I20260619 14:32:58.705516 11844 leader_election.cc:302] T 00000000000000000000000000000000 P 1171af5a19224b379c9e709ad88d37a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1171af5a19224b379c9e709ad88d37a3; no voters:
I20260619 14:32:58.705804 11844 leader_election.cc:288] T 00000000000000000000000000000000 P 1171af5a19224b379c9e709ad88d37a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:32:58.705945 11849 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1171af5a19224b379c9e709ad88d37a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:32:58.706228 11849 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1171af5a19224b379c9e709ad88d37a3 [term 1 LEADER]: Becoming Leader. State: Replica: 1171af5a19224b379c9e709ad88d37a3, State: Running, Role: LEADER
I20260619 14:32:58.706566 11849 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1171af5a19224b379c9e709ad88d37a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1171af5a19224b379c9e709ad88d37a3" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 37291 } }
I20260619 14:32:58.707144 11844 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1171af5a19224b379c9e709ad88d37a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:32:58.708293 11850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1171af5a19224b379c9e709ad88d37a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1171af5a19224b379c9e709ad88d37a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1171af5a19224b379c9e709ad88d37a3" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 37291 } } }
I20260619 14:32:58.708426 11850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1171af5a19224b379c9e709ad88d37a3 [sys.catalog]: This master's current role is: LEADER
I20260619 14:32:58.708814 11858 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:32:58.708775 11851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1171af5a19224b379c9e709ad88d37a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1171af5a19224b379c9e709ad88d37a3. Latest consensus state: current_term: 1 leader_uuid: "1171af5a19224b379c9e709ad88d37a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1171af5a19224b379c9e709ad88d37a3" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 37291 } } }
I20260619 14:32:58.709579 11851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1171af5a19224b379c9e709ad88d37a3 [sys.catalog]: This master's current role is: LEADER
I20260619 14:32:58.712087 11858 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:32:58.717482 11858 catalog_manager.cc:1347] Generated new cluster ID: 3682a437f49a41e4a1496b38fdd9de15
I20260619 14:32:58.717554 11858 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:32:58.729790 11858 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:32:58.730225 11858 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:32:58.737816 11858 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1171af5a19224b379c9e709ad88d37a3: Generated new TSK 0
I20260619 14:32:58.738608 11858 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:32:58.741745 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.1:0
--local_ip_for_outbound_sockets=127.2.2.1
--webserver_interface=127.2.2.1
--webserver_port=0
--tserver_master_addrs=127.2.2.62:37291
--builtin_ntp_servers=127.2.2.20:38311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 14:32:58.854878 11868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:58.855211 11868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:58.855298 11868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:58.859944 11868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:58.860044 11868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:58.860139 11868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.1
I20260619 14:32:58.865247 11868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:38311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.2.1
--webserver_port=0
--tserver_master_addrs=127.2.2.62:37291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.1
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:32:58.866669 11868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:58.868126 11868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:32:58.878108 11874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:58.878180 11876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:58.878144 11873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:58.878497 11868 server_base.cc:1034] running on GCE node
I20260619 14:32:58.878947 11868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:58.879619 11868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:58.880839 11868 hybrid_clock.cc:648] HybridClock initialized: now 1781879578880816 us; error 43 us; skew 500 ppm
I20260619 14:32:58.883003 11868 webserver.cc:460] Webserver started at http://127.2.2.1:44037/ using document root <none> and password file <none>
I20260619 14:32:58.883646 11868 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:58.883750 11868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:58.884001 11868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:32:58.886258 11868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-0/data/instance:
uuid: "1677e253e5c74c1ba290b2280fb9ae6d"
format_stamp: "Formatted at 2026-06-19 14:32:58 on dist-test-slave-70rm"
server_key: "3de53504ef201506c15e2c8e0686e78a"
server_key_iv: "221018b81f96625f117d0d71ac5f9056"
server_key_version: "encryptionkey@0"
I20260619 14:32:58.886884 11868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-0/wal/instance:
uuid: "1677e253e5c74c1ba290b2280fb9ae6d"
format_stamp: "Formatted at 2026-06-19 14:32:58 on dist-test-slave-70rm"
server_key: "3de53504ef201506c15e2c8e0686e78a"
server_key_iv: "221018b81f96625f117d0d71ac5f9056"
server_key_version: "encryptionkey@0"
I20260619 14:32:58.890897 11868 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 14:32:58.893608 11882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:58.894996 11868 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:32:58.895143 11868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-0/wal
uuid: "1677e253e5c74c1ba290b2280fb9ae6d"
format_stamp: "Formatted at 2026-06-19 14:32:58 on dist-test-slave-70rm"
server_key: "3de53504ef201506c15e2c8e0686e78a"
server_key_iv: "221018b81f96625f117d0d71ac5f9056"
server_key_version: "encryptionkey@0"
I20260619 14:32:58.895268 11868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:58.900035 11868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:58.900666 11868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:32:58.900857 11868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:58.901410 11868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:32:58.902379 11868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:32:58.902449 11868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:58.902519 11868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:32:58.902563 11868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:58.911959 11868 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.1:35721
I20260619 14:32:58.911981 11995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.1:35721 every 8 connection(s)
I20260619 14:32:58.912892 11868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
I20260619 14:32:58.918356 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 11868
I20260619 14:32:58.918486 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-0/wal/instance
I20260619 14:32:58.918852 2056 external_mini_cluster.cc:1428] Setting key 17cf1f2ec50a3f2ceb7406a42caccda0
I20260619 14:32:58.919579 11847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:58.914834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:38619 (local address 127.2.2.62:37291)
0619 14:32:58.914977 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:58.914980 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:58.916110 (+ 1130us) server_negotiation.cc:407] Connection header received
0619 14:32:58.917476 (+ 1366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:58.917479 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:58.917528 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:58.917616 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:58.919078 (+ 1462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:58.919081 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:58.919083 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:32:58.919096 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:58.919117 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:32:58.919125 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:58.919222 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:58.919400 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 14:32:58.919454 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 14:32:58.920977 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.2:0
--local_ip_for_outbound_sockets=127.2.2.2
--webserver_interface=127.2.2.2
--webserver_port=0
--tserver_master_addrs=127.2.2.62:37291
--builtin_ntp_servers=127.2.2.20:38311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 14:32:58.921152 11998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:58.915247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:37291 (local address 127.2.2.1:38619)
0619 14:32:58.915934 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:58.915977 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:58.917248 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:58.917803 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:58.917810 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:58.917906 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:58.918739 (+ 833us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:32:58.918741 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:58.918801 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:32:58.918898 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:58.919233 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:58.919244 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:58.919250 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 14:32:58.919409 (+ 159us) client_negotiation.cc:240] Negotiation successful
0619 14:32:58.920136 (+ 727us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":116,"threads_started":1}
I20260619 14:32:58.922452 11996 heartbeater.cc:347] Connected to a master server at 127.2.2.62:37291
I20260619 14:32:58.922703 11996 heartbeater.cc:464] Registering TS with master...
I20260619 14:32:58.923130 11996 heartbeater.cc:511] Master 127.2.2.62:37291 requested a full tablet report, sending...
I20260619 14:32:58.924790 11808 ts_manager.cc:194] Registered new tserver with Master: 1677e253e5c74c1ba290b2280fb9ae6d (127.2.2.1:35721)
W20260619 14:32:59.031129 11999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:59.031431 11999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:59.031473 11999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:59.035503 11999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:59.035558 11999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:59.035614 11999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.2
I20260619 14:32:59.040469 11999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:38311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.2.2
--webserver_port=0
--tserver_master_addrs=127.2.2.62:37291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.2
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:32:59.041518 11999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:59.042892 11999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:32:59.052572 12005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:59.052582 12004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:59.052745 11999 server_base.cc:1034] running on GCE node
W20260619 14:32:59.052820 12007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:59.053337 11999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:59.054024 11999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:59.055222 11999 hybrid_clock.cc:648] HybridClock initialized: now 1781879579055209 us; error 36 us; skew 500 ppm
I20260619 14:32:59.057116 11999 webserver.cc:460] Webserver started at http://127.2.2.2:36179/ using document root <none> and password file <none>
I20260619 14:32:59.057700 11999 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:59.057793 11999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:59.058046 11999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:32:59.060109 11999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-1/data/instance:
uuid: "346fbad714004541916e6782d5a55918"
format_stamp: "Formatted at 2026-06-19 14:32:59 on dist-test-slave-70rm"
server_key: "eca0f73a445c78a40a723f0c854be68b"
server_key_iv: "c3fd2a24a415a89b728149417a4e082d"
server_key_version: "encryptionkey@0"
I20260619 14:32:59.060678 11999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-1/wal/instance:
uuid: "346fbad714004541916e6782d5a55918"
format_stamp: "Formatted at 2026-06-19 14:32:59 on dist-test-slave-70rm"
server_key: "eca0f73a445c78a40a723f0c854be68b"
server_key_iv: "c3fd2a24a415a89b728149417a4e082d"
server_key_version: "encryptionkey@0"
I20260619 14:32:59.064440 11999 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260619 14:32:59.067031 12013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:59.068276 11999 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260619 14:32:59.068418 11999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-1/wal
uuid: "346fbad714004541916e6782d5a55918"
format_stamp: "Formatted at 2026-06-19 14:32:59 on dist-test-slave-70rm"
server_key: "eca0f73a445c78a40a723f0c854be68b"
server_key_iv: "c3fd2a24a415a89b728149417a4e082d"
server_key_version: "encryptionkey@0"
I20260619 14:32:59.068537 11999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:59.073047 11999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:59.073741 11999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:32:59.073949 11999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:59.074561 11999 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:32:59.075644 11999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:32:59.075711 11999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:59.075774 11999 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:32:59.075825 11999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:59.084525 11999 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.2:42823
I20260619 14:32:59.084579 12126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.2:42823 every 8 connection(s)
I20260619 14:32:59.085475 11999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
I20260619 14:32:59.087524 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 11999
I20260619 14:32:59.087646 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-1/wal/instance
I20260619 14:32:59.087968 2056 external_mini_cluster.cc:1428] Setting key c68add106e76528e20581526af61cca1
I20260619 14:32:59.090020 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.3:0
--local_ip_for_outbound_sockets=127.2.2.3
--webserver_interface=127.2.2.3
--webserver_port=0
--tserver_master_addrs=127.2.2.62:37291
--builtin_ntp_servers=127.2.2.20:38311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 14:32:59.092329 11847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:59.087366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:38235 (local address 127.2.2.62:37291)
0619 14:32:59.087519 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:59.087530 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:59.088901 (+ 1371us) server_negotiation.cc:407] Connection header received
0619 14:32:59.090535 (+ 1634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:59.090541 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:59.090587 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:59.090679 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:59.091832 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:59.091838 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:59.091843 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:32:59.091860 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:59.091888 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:32:59.091900 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:59.092007 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:59.092110 (+ 103us) server_negotiation.cc:299] Negotiation successful
0619 14:32:59.092158 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 14:32:59.093147 12129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:59.087926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:37291 (local address 127.2.2.2:38235)
0619 14:32:59.088757 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:59.088814 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:59.090329 (+ 1515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:59.090812 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:59.090818 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:59.090926 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:59.091547 (+ 621us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:32:59.091548 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:59.091592 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:32:59.091671 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:59.091987 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:59.091992 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:59.091995 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 14:32:59.092120 (+ 125us) client_negotiation.cc:240] Negotiation successful
0619 14:32:59.092316 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":150,"threads_started":1}
I20260619 14:32:59.094434 12127 heartbeater.cc:347] Connected to a master server at 127.2.2.62:37291
I20260619 14:32:59.094671 12127 heartbeater.cc:464] Registering TS with master...
I20260619 14:32:59.095093 12127 heartbeater.cc:511] Master 127.2.2.62:37291 requested a full tablet report, sending...
I20260619 14:32:59.096148 11808 ts_manager.cc:194] Registered new tserver with Master: 346fbad714004541916e6782d5a55918 (127.2.2.2:42823)
W20260619 14:32:59.204223 12130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:32:59.204495 12130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:32:59.204536 12130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:32:59.208856 12130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:32:59.208920 12130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:32:59.209005 12130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.3
I20260619 14:32:59.213891 12130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:38311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.2.3
--webserver_port=0
--tserver_master_addrs=127.2.2.62:37291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.3
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:32:59.214995 12130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:32:59.216406 12130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:32:59.226234 12135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:59.226222 12138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:32:59.226222 12136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:32:59.226526 12130 server_base.cc:1034] running on GCE node
I20260619 14:32:59.226950 12130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:32:59.227533 12130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:32:59.228719 12130 hybrid_clock.cc:648] HybridClock initialized: now 1781879579228702 us; error 28 us; skew 500 ppm
I20260619 14:32:59.230772 12130 webserver.cc:460] Webserver started at http://127.2.2.3:42655/ using document root <none> and password file <none>
I20260619 14:32:59.231350 12130 fs_manager.cc:359] Metadata directory not provided
I20260619 14:32:59.231413 12130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:32:59.231581 12130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:32:59.233719 12130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-2/data/instance:
uuid: "4b26d551022a41709dc055efb83b0603"
format_stamp: "Formatted at 2026-06-19 14:32:59 on dist-test-slave-70rm"
server_key: "cb7a831b161cc55f068f23cfb250af94"
server_key_iv: "4483d1139abf2feb2690a7b4412ae9ea"
server_key_version: "encryptionkey@0"
I20260619 14:32:59.234320 12130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-2/wal/instance:
uuid: "4b26d551022a41709dc055efb83b0603"
format_stamp: "Formatted at 2026-06-19 14:32:59 on dist-test-slave-70rm"
server_key: "cb7a831b161cc55f068f23cfb250af94"
server_key_iv: "4483d1139abf2feb2690a7b4412ae9ea"
server_key_version: "encryptionkey@0"
I20260619 14:32:59.238237 12130 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:32:59.241032 12144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:59.242519 12130 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:32:59.242625 12130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-2/wal
uuid: "4b26d551022a41709dc055efb83b0603"
format_stamp: "Formatted at 2026-06-19 14:32:59 on dist-test-slave-70rm"
server_key: "cb7a831b161cc55f068f23cfb250af94"
server_key_iv: "4483d1139abf2feb2690a7b4412ae9ea"
server_key_version: "encryptionkey@0"
I20260619 14:32:59.242725 12130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:32:59.247277 12130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:32:59.247839 12130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:32:59.247974 12130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:32:59.248485 12130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:32:59.249446 12130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:32:59.249491 12130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:59.249527 12130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:32:59.249542 12130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:32:59.258438 12130 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.3:40555
I20260619 14:32:59.258455 12257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.3:40555 every 8 connection(s)
I20260619 14:32:59.259357 12130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
I20260619 14:32:59.265553 11847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:59.261128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.3:34801 (local address 127.2.2.62:37291)
0619 14:32:59.261278 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:59.261282 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:59.262421 (+ 1139us) server_negotiation.cc:407] Connection header received
0619 14:32:59.263755 (+ 1334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:59.263758 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:59.263806 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:59.263914 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:59.265066 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:32:59.265070 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:32:59.265073 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:32:59.265085 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:32:59.265113 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:32:59.265122 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:32:59.265225 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 14:32:59.265369 (+ 144us) server_negotiation.cc:299] Negotiation successful
0619 14:32:59.265415 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 14:32:59.266474 12260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:32:59.261531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:37291 (local address 127.2.2.3:34801)
0619 14:32:59.262265 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0619 14:32:59.262312 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 14:32:59.263537 (+ 1225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:32:59.264083 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:32:59.264091 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:32:59.264182 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:32:59.264800 (+ 618us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:32:59.264802 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:32:59.264843 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:32:59.264911 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:32:59.265206 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:32:59.265214 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:32:59.265220 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 14:32:59.265402 (+ 182us) client_negotiation.cc:240] Negotiation successful
0619 14:32:59.265642 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":127,"threads_started":1}
I20260619 14:32:59.266927 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 12130
I20260619 14:32:59.267067 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879514993456-2056-0/minicluster-data/ts-2/wal/instance
I20260619 14:32:59.267474 2056 external_mini_cluster.cc:1428] Setting key e150a9313c36ef752ca509e5987a85be
I20260619 14:32:59.267930 12258 heartbeater.cc:347] Connected to a master server at 127.2.2.62:37291
I20260619 14:32:59.268199 12258 heartbeater.cc:464] Registering TS with master...
I20260619 14:32:59.268718 12258 heartbeater.cc:511] Master 127.2.2.62:37291 requested a full tablet report, sending...
I20260619 14:32:59.269783 11808 ts_manager.cc:194] Registered new tserver with Master: 4b26d551022a41709dc055efb83b0603 (127.2.2.3:40555)
I20260619 14:32:59.281797 2056 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 14:32:59.284868 11847 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:32:59.283905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54256 (local address 127.2.2.62:37291)
0619 14:32:59.284115 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 14:32:59.284119 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:32:59.284181 (+ 62us) server_negotiation.cc:407] Connection header received
0619 14:32:59.284295 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:32:59.284297 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:32:59.284331 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:32:59.284399 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:32:59.284745 (+ 346us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54256: BlockingRecv error: recv got EOF from 127.0.0.1:54256 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
I20260619 14:32:59.285358 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 11868
I20260619 14:32:59.290779 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 11999
I20260619 14:32:59.296088 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 12130
I20260619 14:32:59.301218 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 11777
2026-06-19T14:32:59Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (811 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:32:59 dist-test-slave-70rm krb5kdc[12268](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:32:59 dist-test-slave-70rm krb5kdc[12268](info): set up 2 sockets
Jun 19 14:32:59 dist-test-slave-70rm krb5kdc[12268](info): commencing operation
krb5kdc: starting...
W20260619 14:33:00.351783 2056 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Jun 19 14:33:00 dist-test-slave-70rm krb5kdc[12268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:33:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:33:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.2.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab.
Entry for principal kudu/127.2.2.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab.
I20260619 14:33:00.475086 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:33927
--webserver_interface=127.2.2.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:39587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.2.62:33927
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:33:00.589432 12281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:33:00.589761 12281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:33:00.589843 12281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:33:00.593894 12281 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:33:00.594012 12281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:33:00.594053 12281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:33:00.594091 12281 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:33:00.594139 12281 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:33:00.599553 12281 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:39587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.2.62:33927
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:33927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.2.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:33:00.600703 12281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:33:00.601745 12281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:33:00.610913 12286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:33:00.610898 12289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:33:00.610898 12281 server_base.cc:1034] running on GCE node
W20260619 14:33:00.610934 12287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:33:00.611821 12281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:33:00.612984 12281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:33:00.614212 12281 hybrid_clock.cc:648] HybridClock initialized: now 1781879580614200 us; error 33 us; skew 500 ppm
Jun 19 14:33:00 dist-test-slave-70rm krb5kdc[12268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879580, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:33:00.617065 12281 init.cc:377] Logged in from keytab as kudu/127.2.2.62@KRBTEST.COM (short username kudu)
I20260619 14:33:00.618033 12281 webserver.cc:460] Webserver started at http://127.2.2.62:37891/ using document root <none> and password file <none>
I20260619 14:33:00.618643 12281 fs_manager.cc:359] Metadata directory not provided
I20260619 14:33:00.618743 12281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:33:00.618988 12281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:33:00.621095 12281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/master-0/data/instance:
uuid: "177424109442400da960fcff8604a0bb"
format_stamp: "Formatted at 2026-06-19 14:33:00 on dist-test-slave-70rm"
server_key: "1d0400d0ffd829f56c70c9e4ebe06839"
server_key_iv: "ab5a912151c7e3272e262fbfa54f8636"
server_key_version: "encryptionkey@0"
I20260619 14:33:00.621666 12281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/master-0/wal/instance:
uuid: "177424109442400da960fcff8604a0bb"
format_stamp: "Formatted at 2026-06-19 14:33:00 on dist-test-slave-70rm"
server_key: "1d0400d0ffd829f56c70c9e4ebe06839"
server_key_iv: "ab5a912151c7e3272e262fbfa54f8636"
server_key_version: "encryptionkey@0"
I20260619 14:33:00.625680 12281 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 14:33:00.628307 12296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:33:00.629588 12281 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 14:33:00.629762 12281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/master-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/master-0/wal
uuid: "177424109442400da960fcff8604a0bb"
format_stamp: "Formatted at 2026-06-19 14:33:00 on dist-test-slave-70rm"
server_key: "1d0400d0ffd829f56c70c9e4ebe06839"
server_key_iv: "ab5a912151c7e3272e262fbfa54f8636"
server_key_version: "encryptionkey@0"
I20260619 14:33:00.629901 12281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:33:00.657334 12281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:33:00.660477 12281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:33:00.660702 12281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:33:00.668128 12281 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.62:33927
I20260619 14:33:00.668429 12348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.62:33927 every 8 connection(s)
I20260619 14:33:00.669269 12281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
I20260619 14:33:00.671860 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 12281
I20260619 14:33:00.672045 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/master-0/wal/instance
I20260619 14:33:00.672344 2056 external_mini_cluster.cc:1428] Setting key 372e2afad5f203df465ae3cec1ca4213
I20260619 14:33:00.672438 12349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:33:00.679503 12349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 177424109442400da960fcff8604a0bb: Bootstrap starting.
Jun 19 14:33:00 dist-test-slave-70rm krb5kdc[12268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:33:00.682217 12349 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 177424109442400da960fcff8604a0bb: Neither blocks nor log segments found. Creating new log.
I20260619 14:33:00.683135 12349 log.cc:824] T 00000000000000000000000000000000 P 177424109442400da960fcff8604a0bb: Log is configured to *not* fsync() on all Append() calls
I20260619 14:33:00.685528 12349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 177424109442400da960fcff8604a0bb: No bootstrap required, opened a new log
I20260619 14:33:00.688198 12352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:33:00.673737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40434 (local address 127.2.2.62:33927)
0619 14:33:00.674300 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0619 14:33:00.674318 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 14:33:00.674359 (+ 41us) server_negotiation.cc:407] Connection header received
0619 14:33:00.675047 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:33:00.675065 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:33:00.675445 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:33:00.675851 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:33:00.676691 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:33:00.677743 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:33:00.678703 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:33:00.679221 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:33:00.679398 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:33:00.682350 (+ 2952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:33:00.682363 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:33:00.682380 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:33:00.682413 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:33:00.684626 (+ 2213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:33:00.685157 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:33:00.685165 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:33:00.685174 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:33:00.685327 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:33:00.685720 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:33:00.685723 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:33:00.685725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:33:00.686309 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:33:00.686501 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0619 14:33:00.686791 (+ 290us) server_negotiation.cc:299] Negotiation successful
0619 14:33:00.687098 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":125,"threads_started":1}
I20260619 14:33:00.690243 12349 raft_consensus.cc:348] T 00000000000000000000000000000000 P 177424109442400da960fcff8604a0bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "177424109442400da960fcff8604a0bb" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 33927 } }
I20260619 14:33:00.690541 12349 raft_consensus.cc:374] T 00000000000000000000000000000000 P 177424109442400da960fcff8604a0bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:33:00.690634 12349 raft_consensus.cc:729] T 00000000000000000000000000000000 P 177424109442400da960fcff8604a0bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 177424109442400da960fcff8604a0bb, State: Initialized, Role: FOLLOWER
I20260619 14:33:00.691133 12349 consensus_queue.cc:260] T 00000000000000000000000000000000 P 177424109442400da960fcff8604a0bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "177424109442400da960fcff8604a0bb" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 33927 } }
I20260619 14:33:00.691298 12349 raft_consensus.cc:388] T 00000000000000000000000000000000 P 177424109442400da960fcff8604a0bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:33:00.691365 12349 raft_consensus.cc:482] T 00000000000000000000000000000000 P 177424109442400da960fcff8604a0bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:33:00.691490 12349 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 177424109442400da960fcff8604a0bb [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:33:00.692458 12349 raft_consensus.cc:504] T 00000000000000000000000000000000 P 177424109442400da960fcff8604a0bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "177424109442400da960fcff8604a0bb" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 33927 } }
I20260619 14:33:00.692821 12349 leader_election.cc:302] T 00000000000000000000000000000000 P 177424109442400da960fcff8604a0bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 177424109442400da960fcff8604a0bb; no voters:
I20260619 14:33:00.693184 12349 leader_election.cc:288] T 00000000000000000000000000000000 P 177424109442400da960fcff8604a0bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:33:00.693331 12354 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 177424109442400da960fcff8604a0bb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:33:00.693576 12354 raft_consensus.cc:686] T 00000000000000000000000000000000 P 177424109442400da960fcff8604a0bb [term 1 LEADER]: Becoming Leader. State: Replica: 177424109442400da960fcff8604a0bb, State: Running, Role: LEADER
I20260619 14:33:00.693929 12354 consensus_queue.cc:237] T 00000000000000000000000000000000 P 177424109442400da960fcff8604a0bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "177424109442400da960fcff8604a0bb" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 33927 } }
I20260619 14:33:00.694538 12349 sys_catalog.cc:564] T 00000000000000000000000000000000 P 177424109442400da960fcff8604a0bb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:33:00.695503 12356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 177424109442400da960fcff8604a0bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 177424109442400da960fcff8604a0bb. Latest consensus state: current_term: 1 leader_uuid: "177424109442400da960fcff8604a0bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "177424109442400da960fcff8604a0bb" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 33927 } } }
I20260619 14:33:00.695643 12356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 177424109442400da960fcff8604a0bb [sys.catalog]: This master's current role is: LEADER
I20260619 14:33:00.696036 12355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 177424109442400da960fcff8604a0bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "177424109442400da960fcff8604a0bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "177424109442400da960fcff8604a0bb" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 33927 } } }
I20260619 14:33:00.696174 12355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 177424109442400da960fcff8604a0bb [sys.catalog]: This master's current role is: LEADER
I20260619 14:33:00.696794 12363 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:33:00.699702 12363 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:33:00.704948 12363 catalog_manager.cc:1347] Generated new cluster ID: 57fff424881341cf81989cfe203697b7
I20260619 14:33:00.705015 12363 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:33:00.719445 12363 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:33:00.720431 12363 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:33:00.729789 12363 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 177424109442400da960fcff8604a0bb: Generated new TSK 0
I20260619 14:33:00.730607 12363 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.2.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab.
Entry for principal kudu/127.2.2.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab.
I20260619 14:33:00.761751 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.1:0
--local_ip_for_outbound_sockets=127.2.2.1
--webserver_interface=127.2.2.1
--webserver_port=0
--tserver_master_addrs=127.2.2.62:33927
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab
--principal=kudu/127.2.2.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:39587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:33:00.879925 12375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:33:00.880240 12375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:33:00.880287 12375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:33:00.884485 12375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:33:00.884562 12375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:33:00.884624 12375 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.1
I20260619 14:33:00.890281 12375 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:39587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab
--principal=kudu/127.2.2.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.2.1
--webserver_port=0
--tserver_master_addrs=127.2.2.62:33927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.1
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:33:00.891471 12375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:33:00.892557 12375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:33:00.902912 12383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:33:00.902912 12380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:33:00.902978 12375 server_base.cc:1034] running on GCE node
W20260619 14:33:00.902916 12381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:33:00.903647 12375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:33:00.904352 12375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:33:00.905575 12375 hybrid_clock.cc:648] HybridClock initialized: now 1781879580905555 us; error 33 us; skew 500 ppm
Jun 19 14:33:00 dist-test-slave-70rm krb5kdc[12268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879580, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:33:00.908916 12375 init.cc:377] Logged in from keytab as kudu/127.2.2.1@KRBTEST.COM (short username kudu)
I20260619 14:33:00.909945 12375 webserver.cc:460] Webserver started at http://127.2.2.1:39043/ using document root <none> and password file <none>
I20260619 14:33:00.910600 12375 fs_manager.cc:359] Metadata directory not provided
I20260619 14:33:00.910697 12375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:33:00.910950 12375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:33:00.913069 12375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/ts-0/data/instance:
uuid: "d72375b43f114df5960175eaac61229d"
format_stamp: "Formatted at 2026-06-19 14:33:00 on dist-test-slave-70rm"
server_key: "633da0fe181890d480455c0bda330e1c"
server_key_iv: "14ff2c24a9f57ac1b5b58f6b3e96e6b7"
server_key_version: "encryptionkey@0"
I20260619 14:33:00.913676 12375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/ts-0/wal/instance:
uuid: "d72375b43f114df5960175eaac61229d"
format_stamp: "Formatted at 2026-06-19 14:33:00 on dist-test-slave-70rm"
server_key: "633da0fe181890d480455c0bda330e1c"
server_key_iv: "14ff2c24a9f57ac1b5b58f6b3e96e6b7"
server_key_version: "encryptionkey@0"
I20260619 14:33:00.917663 12375 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 14:33:00.920506 12390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:33:00.921893 12375 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 14:33:00.922044 12375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/ts-0/wal
uuid: "d72375b43f114df5960175eaac61229d"
format_stamp: "Formatted at 2026-06-19 14:33:00 on dist-test-slave-70rm"
server_key: "633da0fe181890d480455c0bda330e1c"
server_key_iv: "14ff2c24a9f57ac1b5b58f6b3e96e6b7"
server_key_version: "encryptionkey@0"
I20260619 14:33:00.922230 12375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:33:00.949397 12375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:33:00.953047 12375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:33:00.953284 12375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:33:00.953935 12375 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:33:00.954878 12375 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:33:00.954962 12375 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:33:00.955052 12375 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:33:00.955106 12375 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:33:00.964416 12375 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.1:36679
I20260619 14:33:00.964439 12503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.1:36679 every 8 connection(s)
I20260619 14:33:00.965384 12375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
I20260619 14:33:00.969255 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 12375
I20260619 14:33:00.969353 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/ts-0/wal/instance
I20260619 14:33:00.969653 2056 external_mini_cluster.cc:1428] Setting key 49178ad43232bafeaa6f7621f0192436
Jun 19 14:33:00 dist-test-slave-70rm krb5kdc[12268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879580, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.1@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.2.2@KRBTEST.COM; defaulting to no policy
I20260619 14:33:00.980569 12352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:33:00.967349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:42883 (local address 127.2.2.62:33927)
0619 14:33:00.967491 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 14:33:00.967495 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:33:00.968319 (+ 824us) server_negotiation.cc:407] Connection header received
0619 14:33:00.969756 (+ 1437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:33:00.969761 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:33:00.969819 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:33:00.969924 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:33:00.971327 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:33:00.972152 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:33:00.973358 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:33:00.973806 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:33:00.973950 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:33:00.976560 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:33:00.976566 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:33:00.976572 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:33:00.976611 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:33:00.978401 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:33:00.978948 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:33:00.978954 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:33:00.978956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:33:00.979053 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:33:00.979416 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:33:00.979422 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:33:00.979427 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:33:00.979687 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:33:00.979789 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 14:33:00.980325 (+ 536us) server_negotiation.cc:299] Negotiation successful
0619 14:33:00.980438 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 14:33:00.981478 12506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:33:00.967732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:33927 (local address 127.2.2.1:42883)
0619 14:33:00.968166 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0619 14:33:00.968209 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 14:33:00.969527 (+ 1318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:33:00.970112 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:33:00.970122 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:33:00.970570 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:33:00.971182 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:33:00.971200 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:33:00.972282 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:33:00.972285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:33:00.973215 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:33:00.973224 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:33:00.973918 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:33:00.973968 (+ 50us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:33:00.974164 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:33:00.974831 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:33:00.974848 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:33:00.976360 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:33:00.978527 (+ 2167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:33:00.978533 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:33:00.978547 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:33:00.978833 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:33:00.979155 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:33:00.979157 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:33:00.979159 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:33:00.979295 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:33:00.979785 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:33:00.979790 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:33:00.980093 (+ 303us) client_negotiation.cc:769] Sending connection context
0619 14:33:00.980256 (+ 163us) client_negotiation.cc:240] Negotiation successful
0619 14:33:00.980562 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":110,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.2@KRBTEST.COM" created.
I20260619 14:33:00.982985 12504 heartbeater.cc:347] Connected to a master server at 127.2.2.62:33927
I20260619 14:33:00.983258 12504 heartbeater.cc:464] Registering TS with master...
I20260619 14:33:00.983768 12504 heartbeater.cc:511] Master 127.2.2.62:33927 requested a full tablet report, sending...
I20260619 14:33:00.985445 12313 ts_manager.cc:194] Registered new tserver with Master: d72375b43f114df5960175eaac61229d (127.2.2.1:36679)
I20260619 14:33:00.986788 12313 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.2.1@KRBTEST.COM'} at 127.2.2.1:42883
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab.
Entry for principal kudu/127.2.2.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab.
I20260619 14:33:00.998816 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.2:0
--local_ip_for_outbound_sockets=127.2.2.2
--webserver_interface=127.2.2.2
--webserver_port=0
--tserver_master_addrs=127.2.2.62:33927
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab
--principal=kudu/127.2.2.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:39587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:33:01.112407 12509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:33:01.112763 12509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:33:01.112838 12509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:33:01.117765 12509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:33:01.117868 12509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:33:01.117975 12509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.2
I20260619 14:33:01.123617 12509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:39587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab
--principal=kudu/127.2.2.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.2.2
--webserver_port=0
--tserver_master_addrs=127.2.2.62:33927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.2
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:33:01.124871 12509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:33:01.126013 12509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:33:01.136267 12515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:33:01.136267 12517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:33:01.136296 12514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:33:01.136775 12509 server_base.cc:1034] running on GCE node
I20260619 14:33:01.137141 12509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:33:01.137805 12509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:33:01.138999 12509 hybrid_clock.cc:648] HybridClock initialized: now 1781879581138984 us; error 36 us; skew 500 ppm
Jun 19 14:33:01 dist-test-slave-70rm krb5kdc[12268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879581, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:33:01.141848 12509 init.cc:377] Logged in from keytab as kudu/127.2.2.2@KRBTEST.COM (short username kudu)
I20260619 14:33:01.142860 12509 webserver.cc:460] Webserver started at http://127.2.2.2:33139/ using document root <none> and password file <none>
I20260619 14:33:01.143385 12509 fs_manager.cc:359] Metadata directory not provided
I20260619 14:33:01.143445 12509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:33:01.143623 12509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:33:01.145905 12509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/ts-1/data/instance:
uuid: "24036d7afbfa4100855591e1dd40f7af"
format_stamp: "Formatted at 2026-06-19 14:33:01 on dist-test-slave-70rm"
server_key: "4ffec03d63ac33cbe2e636549796c8f5"
server_key_iv: "a4ea45cbc71a2d5097bf5c21ff26efc0"
server_key_version: "encryptionkey@0"
I20260619 14:33:01.146497 12509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/ts-1/wal/instance:
uuid: "24036d7afbfa4100855591e1dd40f7af"
format_stamp: "Formatted at 2026-06-19 14:33:01 on dist-test-slave-70rm"
server_key: "4ffec03d63ac33cbe2e636549796c8f5"
server_key_iv: "a4ea45cbc71a2d5097bf5c21ff26efc0"
server_key_version: "encryptionkey@0"
I20260619 14:33:01.150276 12509 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:33:01.152768 12524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:33:01.154088 12509 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:33:01.154302 12509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/ts-1/wal
uuid: "24036d7afbfa4100855591e1dd40f7af"
format_stamp: "Formatted at 2026-06-19 14:33:01 on dist-test-slave-70rm"
server_key: "4ffec03d63ac33cbe2e636549796c8f5"
server_key_iv: "a4ea45cbc71a2d5097bf5c21ff26efc0"
server_key_version: "encryptionkey@0"
I20260619 14:33:01.154412 12509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:33:01.188894 12509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:33:01.192655 12509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:33:01.192836 12509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:33:01.193411 12509 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:33:01.194489 12509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:33:01.194567 12509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:33:01.194643 12509 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:33:01.194670 12509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:33:01.204792 12509 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.2:43235
I20260619 14:33:01.204813 12637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.2:43235 every 8 connection(s)
I20260619 14:33:01.205843 12509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
I20260619 14:33:01.215703 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 12509
Jun 19 14:33:01 dist-test-slave-70rm krb5kdc[12268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879581, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.2@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:33:01.215857 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/ts-1/wal/instance
I20260619 14:33:01.216235 2056 external_mini_cluster.cc:1428] Setting key 65d4ea17498619e1c8cc1c7ebdbce2df
I20260619 14:33:01.221127 12352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:33:01.207833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:60171 (local address 127.2.2.62:33927)
0619 14:33:01.207980 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 14:33:01.207984 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:33:01.208916 (+ 932us) server_negotiation.cc:407] Connection header received
0619 14:33:01.210392 (+ 1476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:33:01.210396 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:33:01.210445 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:33:01.210527 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:33:01.211780 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:33:01.212335 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:33:01.213510 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:33:01.213733 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:33:01.213818 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:33:01.216717 (+ 2899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:33:01.216725 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:33:01.216728 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:33:01.216770 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:33:01.218607 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:33:01.219321 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:33:01.219324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:33:01.219325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:33:01.219400 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:33:01.219809 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:33:01.219812 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:33:01.219833 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:33:01.220095 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:33:01.220202 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 14:33:01.220758 (+ 556us) server_negotiation.cc:299] Negotiation successful
0619 14:33:01.220909 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 14:33:01.222172 12640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:33:01.208263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:33927 (local address 127.2.2.2:60171)
0619 14:33:01.208748 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0619 14:33:01.208795 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 14:33:01.210049 (+ 1254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:33:01.210708 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:33:01.210715 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:33:01.211102 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:33:01.211629 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:33:01.211644 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:33:01.212458 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:33:01.212461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:33:01.213399 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:33:01.213408 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:33:01.213826 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:33:01.213875 (+ 49us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:33:01.214022 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:33:01.214714 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:33:01.214731 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:33:01.216448 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:33:01.218756 (+ 2308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:33:01.218766 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:33:01.218787 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:33:01.219219 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:33:01.219493 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:33:01.219499 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:33:01.219502 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:33:01.219707 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:33:01.220232 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:33:01.220237 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:33:01.220506 (+ 269us) client_negotiation.cc:769] Sending connection context
0619 14:33:01.220683 (+ 177us) client_negotiation.cc:240] Negotiation successful
0619 14:33:01.220965 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":124,"threads_started":1}
I20260619 14:33:01.223467 12638 heartbeater.cc:347] Connected to a master server at 127.2.2.62:33927
I20260619 14:33:01.223760 12638 heartbeater.cc:464] Registering TS with master...
I20260619 14:33:01.224455 12638 heartbeater.cc:511] Master 127.2.2.62:33927 requested a full tablet report, sending...
I20260619 14:33:01.225718 12313 ts_manager.cc:194] Registered new tserver with Master: 24036d7afbfa4100855591e1dd40f7af (127.2.2.2:43235)
I20260619 14:33:01.226397 12313 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.2.2@KRBTEST.COM'} at 127.2.2.2:60171
WARNING: no policy specified for kudu/127.2.2.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab.
Entry for principal kudu/127.2.2.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab.
I20260619 14:33:01.245975 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.3:0
--local_ip_for_outbound_sockets=127.2.2.3
--webserver_interface=127.2.2.3
--webserver_port=0
--tserver_master_addrs=127.2.2.62:33927
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab
--principal=kudu/127.2.2.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:39587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:33:01.373451 12643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:33:01.373745 12643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:33:01.373824 12643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:33:01.378053 12643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:33:01.378185 12643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:33:01.378259 12643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.3
I20260619 14:33:01.383671 12643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:39587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab
--principal=kudu/127.2.2.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.2.3
--webserver_port=0
--tserver_master_addrs=127.2.2.62:33927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.3
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:33:01.384829 12643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:33:01.385926 12643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:33:01.396286 12648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:33:01.396508 12643 server_base.cc:1034] running on GCE node
W20260619 14:33:01.396286 12649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:33:01.396289 12651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:33:01.397064 12643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:33:01.397779 12643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:33:01.398945 12643 hybrid_clock.cc:648] HybridClock initialized: now 1781879581398925 us; error 32 us; skew 500 ppm
Jun 19 14:33:01 dist-test-slave-70rm krb5kdc[12268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879581, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:33:01.401891 12643 init.cc:377] Logged in from keytab as kudu/127.2.2.3@KRBTEST.COM (short username kudu)
I20260619 14:33:01.402885 12643 webserver.cc:460] Webserver started at http://127.2.2.3:38013/ using document root <none> and password file <none>
I20260619 14:33:01.403465 12643 fs_manager.cc:359] Metadata directory not provided
I20260619 14:33:01.403527 12643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:33:01.403793 12643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:33:01.405885 12643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/ts-2/data/instance:
uuid: "be04ed08fcfc475eb48a39866616ac48"
format_stamp: "Formatted at 2026-06-19 14:33:01 on dist-test-slave-70rm"
server_key: "508aa95842bfc3f179dbf401fae007f4"
server_key_iv: "cfea4fc874a804d5a6999c52f49add52"
server_key_version: "encryptionkey@0"
I20260619 14:33:01.406524 12643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/ts-2/wal/instance:
uuid: "be04ed08fcfc475eb48a39866616ac48"
format_stamp: "Formatted at 2026-06-19 14:33:01 on dist-test-slave-70rm"
server_key: "508aa95842bfc3f179dbf401fae007f4"
server_key_iv: "cfea4fc874a804d5a6999c52f49add52"
server_key_version: "encryptionkey@0"
I20260619 14:33:01.410511 12643 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 14:33:01.413342 12658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:33:01.414726 12643 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:33:01.414877 12643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/ts-2/wal
uuid: "be04ed08fcfc475eb48a39866616ac48"
format_stamp: "Formatted at 2026-06-19 14:33:01 on dist-test-slave-70rm"
server_key: "508aa95842bfc3f179dbf401fae007f4"
server_key_iv: "cfea4fc874a804d5a6999c52f49add52"
server_key_version: "encryptionkey@0"
I20260619 14:33:01.415007 12643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:33:01.444193 12643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:33:01.448436 12643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:33:01.448683 12643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:33:01.449348 12643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:33:01.450294 12643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:33:01.450372 12643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:33:01.450451 12643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:33:01.450508 12643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:33:01.460723 12643 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.3:43275
I20260619 14:33:01.460743 12771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.3:43275 every 8 connection(s)
I20260619 14:33:01.461720 12643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
I20260619 14:33:01.463171 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 12643
I20260619 14:33:01.463318 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/ts-2/wal/instance
I20260619 14:33:01.463670 2056 external_mini_cluster.cc:1428] Setting key 7aa083726895e9db53f1de2bd0ca2dde
Jun 19 14:33:01 dist-test-slave-70rm krb5kdc[12268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879581, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.3@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:33:01.477535 12352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:33:01.463773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.3:39883 (local address 127.2.2.62:33927)
0619 14:33:01.463915 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 14:33:01.463919 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:33:01.465442 (+ 1523us) server_negotiation.cc:407] Connection header received
0619 14:33:01.467055 (+ 1613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:33:01.467058 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:33:01.467112 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:33:01.467200 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:33:01.468456 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:33:01.469010 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:33:01.470355 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:33:01.470582 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:33:01.470679 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:33:01.473465 (+ 2786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:33:01.473469 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:33:01.473471 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:33:01.473499 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:33:01.475138 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:33:01.475779 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:33:01.475783 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:33:01.475784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:33:01.475887 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:33:01.476240 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:33:01.476242 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:33:01.476243 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:33:01.476465 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:33:01.476541 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 14:33:01.477222 (+ 681us) server_negotiation.cc:299] Negotiation successful
0619 14:33:01.477372 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 14:33:01.478571 12774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:33:01.464477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:33927 (local address 127.2.2.3:39883)
0619 14:33:01.465269 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0619 14:33:01.465333 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 14:33:01.466818 (+ 1485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:33:01.467390 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:33:01.467397 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:33:01.467779 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:33:01.468293 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:33:01.468307 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:33:01.469151 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:33:01.469183 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:33:01.470104 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:33:01.470113 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:33:01.470704 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:33:01.470755 (+ 51us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:33:01.470934 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:33:01.471579 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:33:01.471596 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:33:01.473255 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:33:01.475320 (+ 2065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:33:01.475326 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:33:01.475341 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:33:01.475640 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:33:01.475989 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:33:01.475992 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:33:01.475993 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:33:01.476151 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:33:01.476571 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:33:01.476578 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:33:01.476881 (+ 303us) client_negotiation.cc:769] Sending connection context
0619 14:33:01.477097 (+ 216us) client_negotiation.cc:240] Negotiation successful
0619 14:33:01.477489 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":165,"threads_started":1}
I20260619 14:33:01.479696 12772 heartbeater.cc:347] Connected to a master server at 127.2.2.62:33927
I20260619 14:33:01.479951 12772 heartbeater.cc:464] Registering TS with master...
I20260619 14:33:01.480489 12772 heartbeater.cc:511] Master 127.2.2.62:33927 requested a full tablet report, sending...
I20260619 14:33:01.481527 12313 ts_manager.cc:194] Registered new tserver with Master: be04ed08fcfc475eb48a39866616ac48 (127.2.2.3:43275)
I20260619 14:33:01.482113 12313 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.2.3@KRBTEST.COM'} at 127.2.2.3:39883
I20260619 14:33:01.490197 2056 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:33:01.499495 12352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:33:01.492207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40444 (local address 127.2.2.62:33927)
0619 14:33:01.492334 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 14:33:01.492337 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:33:01.492436 (+ 99us) server_negotiation.cc:407] Connection header received
0619 14:33:01.492537 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:33:01.492539 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:33:01.492577 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:33:01.492650 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:33:01.493330 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:33:01.493841 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:33:01.494827 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:33:01.495044 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:33:01.495127 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:33:01.496659 (+ 1532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:33:01.496662 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:33:01.496664 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:33:01.496700 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:33:01.497995 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:33:01.498413 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:33:01.498416 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:33:01.498417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:33:01.498493 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:33:01.498703 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:33:01.498705 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:33:01.498706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:33:01.498908 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:33:01.499019 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 14:33:01.499279 (+ 260us) server_negotiation.cc:299] Negotiation successful
0619 14:33:01.499373 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 14:33:01.502650 12313 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40444:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:33:01.505748 12313 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:33:01.521185 12788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:33:01.515472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53374 (local address 127.2.2.3:43275)
0619 14:33:01.515912 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0619 14:33:01.515925 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:33:01.516161 (+ 236us) server_negotiation.cc:407] Connection header received
0619 14:33:01.516313 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:33:01.516318 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:33:01.516504 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:33:01.516665 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:33:01.516883 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:33:01.518012 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:33:01.519704 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:33:01.520352 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:33:01.520480 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:33:01.520613 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 14:33:01.520915 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 14:33:01.521019 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":132,"threads_started":1}
I20260619 14:33:01.521173 12786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:33:01.514611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.2:43235 (local address 127.0.0.1:39252)
0619 14:33:01.515506 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0619 14:33:01.515562 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 14:33:01.515692 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:33:01.516260 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:33:01.516266 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:33:01.516288 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:33:01.516420 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:33:01.516426 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:33:01.517629 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:33:01.517635 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:33:01.518921 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:33:01.518932 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:33:01.520603 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:33:01.520606 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:33:01.520719 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:33:01.520791 (+ 72us) client_negotiation.cc:769] Sending connection context
0619 14:33:01.520889 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 14:33:01.521027 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"mutex_wait_us":64,"thread_start_us":96,"threads_started":1}
I20260619 14:33:01.521188 12787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:33:01.515272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50976 (local address 127.2.2.1:36679)
0619 14:33:01.515980 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0619 14:33:01.515986 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:33:01.516193 (+ 207us) server_negotiation.cc:407] Connection header received
0619 14:33:01.516398 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:33:01.516404 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:33:01.516624 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:33:01.516741 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:33:01.517130 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:33:01.518241 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:33:01.519729 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:33:01.520349 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:33:01.520470 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:33:01.520605 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 14:33:01.520943 (+ 338us) server_negotiation.cc:299] Negotiation successful
0619 14:33:01.521040 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":129,"threads_started":1}
I20260619 14:33:01.521179 12785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:33:01.514814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.3:43275 (local address 127.0.0.1:53374)
0619 14:33:01.516047 (+ 1233us) negotiation.cc:107] Waiting for socket to connect
0619 14:33:01.516060 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:33:01.516172 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:33:01.516648 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:33:01.516651 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:33:01.516667 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:33:01.516764 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:33:01.516772 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:33:01.518179 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:33:01.518182 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:33:01.519575 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:33:01.519585 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:33:01.520482 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:33:01.520485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:33:01.520708 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:33:01.520765 (+ 57us) client_negotiation.cc:769] Sending connection context
0619 14:33:01.520870 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 14:33:01.521033 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1172,"mutex_wait_us":182,"thread_start_us":87,"threads_started":1}
I20260619 14:33:01.521633 12784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:33:01.514752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39252 (local address 127.2.2.2:43235)
0619 14:33:01.515727 (+ 975us) server_negotiation.cc:206] Beginning negotiation
0619 14:33:01.515740 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:33:01.515766 (+ 26us) server_negotiation.cc:407] Connection header received
0619 14:33:01.515846 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:33:01.515851 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:33:01.516069 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:33:01.516272 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:33:01.516540 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:33:01.517499 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:33:01.519081 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:33:01.519625 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:33:01.519755 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:33:01.519921 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 14:33:01.521377 (+ 1456us) server_negotiation.cc:299] Negotiation successful
0619 14:33:01.521498 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":874,"thread_start_us":113,"threads_started":1}
I20260619 14:33:01.521946 12783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:33:01.514926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.1:36679 (local address 127.0.0.1:50976)
0619 14:33:01.516110 (+ 1184us) negotiation.cc:107] Waiting for socket to connect
0619 14:33:01.516127 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:33:01.516259 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:33:01.516914 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:33:01.516917 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:33:01.516933 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:33:01.517013 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:33:01.517019 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:33:01.518354 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:33:01.518357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:33:01.519575 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:33:01.519585 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:33:01.520470 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:33:01.520473 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:33:01.520651 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:33:01.520750 (+ 99us) client_negotiation.cc:769] Sending connection context
0619 14:33:01.521754 (+ 1004us) client_negotiation.cc:240] Negotiation successful
0619 14:33:01.521831 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1126,"mutex_wait_us":110,"thread_start_us":76,"threads_started":1}
I20260619 14:33:01.524327 12438 tablet_service.cc:1467] Processing CreateTablet for tablet b7009ea6c22841f79575e30abbe3ab93 (DEFAULT_TABLE table=test-table [id=d6731a7b60a040208a1c79cdbbd491b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:33:01.524325 12572 tablet_service.cc:1467] Processing CreateTablet for tablet b7009ea6c22841f79575e30abbe3ab93 (DEFAULT_TABLE table=test-table [id=d6731a7b60a040208a1c79cdbbd491b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:33:01.525368 12438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7009ea6c22841f79575e30abbe3ab93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:33:01.525379 12572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7009ea6c22841f79575e30abbe3ab93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:33:01.526952 12706 tablet_service.cc:1467] Processing CreateTablet for tablet b7009ea6c22841f79575e30abbe3ab93 (DEFAULT_TABLE table=test-table [id=d6731a7b60a040208a1c79cdbbd491b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:33:01.528010 12706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7009ea6c22841f79575e30abbe3ab93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:33:01.532382 12789 tablet_bootstrap.cc:492] T b7009ea6c22841f79575e30abbe3ab93 P 24036d7afbfa4100855591e1dd40f7af: Bootstrap starting.
I20260619 14:33:01.533746 12790 tablet_bootstrap.cc:492] T b7009ea6c22841f79575e30abbe3ab93 P d72375b43f114df5960175eaac61229d: Bootstrap starting.
I20260619 14:33:01.534830 12789 tablet_bootstrap.cc:654] T b7009ea6c22841f79575e30abbe3ab93 P 24036d7afbfa4100855591e1dd40f7af: Neither blocks nor log segments found. Creating new log.
I20260619 14:33:01.535323 12791 tablet_bootstrap.cc:492] T b7009ea6c22841f79575e30abbe3ab93 P be04ed08fcfc475eb48a39866616ac48: Bootstrap starting.
I20260619 14:33:01.536029 12790 tablet_bootstrap.cc:654] T b7009ea6c22841f79575e30abbe3ab93 P d72375b43f114df5960175eaac61229d: Neither blocks nor log segments found. Creating new log.
I20260619 14:33:01.536216 12789 log.cc:824] T b7009ea6c22841f79575e30abbe3ab93 P 24036d7afbfa4100855591e1dd40f7af: Log is configured to *not* fsync() on all Append() calls
I20260619 14:33:01.536755 12790 log.cc:824] T b7009ea6c22841f79575e30abbe3ab93 P d72375b43f114df5960175eaac61229d: Log is configured to *not* fsync() on all Append() calls
I20260619 14:33:01.537585 12791 tablet_bootstrap.cc:654] T b7009ea6c22841f79575e30abbe3ab93 P be04ed08fcfc475eb48a39866616ac48: Neither blocks nor log segments found. Creating new log.
I20260619 14:33:01.538585 12790 tablet_bootstrap.cc:492] T b7009ea6c22841f79575e30abbe3ab93 P d72375b43f114df5960175eaac61229d: No bootstrap required, opened a new log
I20260619 14:33:01.538585 12789 tablet_bootstrap.cc:492] T b7009ea6c22841f79575e30abbe3ab93 P 24036d7afbfa4100855591e1dd40f7af: No bootstrap required, opened a new log
I20260619 14:33:01.538722 12791 log.cc:824] T b7009ea6c22841f79575e30abbe3ab93 P be04ed08fcfc475eb48a39866616ac48: Log is configured to *not* fsync() on all Append() calls
I20260619 14:33:01.538789 12790 ts_tablet_manager.cc:1397] T b7009ea6c22841f79575e30abbe3ab93 P d72375b43f114df5960175eaac61229d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:33:01.538789 12789 ts_tablet_manager.cc:1397] T b7009ea6c22841f79575e30abbe3ab93 P 24036d7afbfa4100855591e1dd40f7af: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260619 14:33:01.540362 12791 tablet_bootstrap.cc:492] T b7009ea6c22841f79575e30abbe3ab93 P be04ed08fcfc475eb48a39866616ac48: No bootstrap required, opened a new log
I20260619 14:33:01.540562 12791 ts_tablet_manager.cc:1397] T b7009ea6c22841f79575e30abbe3ab93 P be04ed08fcfc475eb48a39866616ac48: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260619 14:33:01.543043 12789 raft_consensus.cc:348] T b7009ea6c22841f79575e30abbe3ab93 P 24036d7afbfa4100855591e1dd40f7af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24036d7afbfa4100855591e1dd40f7af" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 43235 } } peers { permanent_uuid: "be04ed08fcfc475eb48a39866616ac48" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 43275 } } peers { permanent_uuid: "d72375b43f114df5960175eaac61229d" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 36679 } }
I20260619 14:33:01.543388 12789 raft_consensus.cc:374] T b7009ea6c22841f79575e30abbe3ab93 P 24036d7afbfa4100855591e1dd40f7af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:33:01.543248 12790 raft_consensus.cc:348] T b7009ea6c22841f79575e30abbe3ab93 P d72375b43f114df5960175eaac61229d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24036d7afbfa4100855591e1dd40f7af" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 43235 } } peers { permanent_uuid: "be04ed08fcfc475eb48a39866616ac48" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 43275 } } peers { permanent_uuid: "d72375b43f114df5960175eaac61229d" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 36679 } }
I20260619 14:33:01.543445 12789 raft_consensus.cc:729] T b7009ea6c22841f79575e30abbe3ab93 P 24036d7afbfa4100855591e1dd40f7af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24036d7afbfa4100855591e1dd40f7af, State: Initialized, Role: FOLLOWER
I20260619 14:33:01.543466 12790 raft_consensus.cc:374] T b7009ea6c22841f79575e30abbe3ab93 P d72375b43f114df5960175eaac61229d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:33:01.543510 12790 raft_consensus.cc:729] T b7009ea6c22841f79575e30abbe3ab93 P d72375b43f114df5960175eaac61229d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d72375b43f114df5960175eaac61229d, State: Initialized, Role: FOLLOWER
I20260619 14:33:01.543399 12791 raft_consensus.cc:348] T b7009ea6c22841f79575e30abbe3ab93 P be04ed08fcfc475eb48a39866616ac48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24036d7afbfa4100855591e1dd40f7af" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 43235 } } peers { permanent_uuid: "be04ed08fcfc475eb48a39866616ac48" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 43275 } } peers { permanent_uuid: "d72375b43f114df5960175eaac61229d" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 36679 } }
I20260619 14:33:01.543686 12791 raft_consensus.cc:374] T b7009ea6c22841f79575e30abbe3ab93 P be04ed08fcfc475eb48a39866616ac48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:33:01.543753 12791 raft_consensus.cc:729] T b7009ea6c22841f79575e30abbe3ab93 P be04ed08fcfc475eb48a39866616ac48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be04ed08fcfc475eb48a39866616ac48, State: Initialized, Role: FOLLOWER
I20260619 14:33:01.543932 12790 consensus_queue.cc:260] T b7009ea6c22841f79575e30abbe3ab93 P d72375b43f114df5960175eaac61229d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24036d7afbfa4100855591e1dd40f7af" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 43235 } } peers { permanent_uuid: "be04ed08fcfc475eb48a39866616ac48" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 43275 } } peers { permanent_uuid: "d72375b43f114df5960175eaac61229d" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 36679 } }
I20260619 14:33:01.543932 12789 consensus_queue.cc:260] T b7009ea6c22841f79575e30abbe3ab93 P 24036d7afbfa4100855591e1dd40f7af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24036d7afbfa4100855591e1dd40f7af" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 43235 } } peers { permanent_uuid: "be04ed08fcfc475eb48a39866616ac48" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 43275 } } peers { permanent_uuid: "d72375b43f114df5960175eaac61229d" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 36679 } }
I20260619 14:33:01.544178 12791 consensus_queue.cc:260] T b7009ea6c22841f79575e30abbe3ab93 P be04ed08fcfc475eb48a39866616ac48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24036d7afbfa4100855591e1dd40f7af" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 43235 } } peers { permanent_uuid: "be04ed08fcfc475eb48a39866616ac48" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 43275 } } peers { permanent_uuid: "d72375b43f114df5960175eaac61229d" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 36679 } }
I20260619 14:33:01.545029 12789 ts_tablet_manager.cc:1428] T b7009ea6c22841f79575e30abbe3ab93 P 24036d7afbfa4100855591e1dd40f7af: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 14:33:01.545029 12790 ts_tablet_manager.cc:1428] T b7009ea6c22841f79575e30abbe3ab93 P d72375b43f114df5960175eaac61229d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 14:33:01.545145 12791 ts_tablet_manager.cc:1428] T b7009ea6c22841f79575e30abbe3ab93 P be04ed08fcfc475eb48a39866616ac48: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260619 14:33:01.545403 12638 heartbeater.cc:503] Master 127.2.2.62:33927 was elected leader, sending a full tablet report...
I20260619 14:33:01.545404 12504 heartbeater.cc:503] Master 127.2.2.62:33927 was elected leader, sending a full tablet report...
I20260619 14:33:01.545500 12772 heartbeater.cc:503] Master 127.2.2.62:33927 was elected leader, sending a full tablet report...
I20260619 14:33:01.551116 12796 raft_consensus.cc:482] T b7009ea6c22841f79575e30abbe3ab93 P d72375b43f114df5960175eaac61229d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:33:01.551321 12796 raft_consensus.cc:504] T b7009ea6c22841f79575e30abbe3ab93 P d72375b43f114df5960175eaac61229d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24036d7afbfa4100855591e1dd40f7af" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 43235 } } peers { permanent_uuid: "be04ed08fcfc475eb48a39866616ac48" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 43275 } } peers { permanent_uuid: "d72375b43f114df5960175eaac61229d" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 36679 } }
I20260619 14:33:01.552296 12796 leader_election.cc:288] T b7009ea6c22841f79575e30abbe3ab93 P d72375b43f114df5960175eaac61229d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 24036d7afbfa4100855591e1dd40f7af (127.2.2.2:43235), be04ed08fcfc475eb48a39866616ac48 (127.2.2.3:43275)
I20260619 14:33:01.556557 12788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:33:01.552642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:54849 (local address 127.2.2.3:43275)
0619 14:33:01.552777 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 14:33:01.552782 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:33:01.553092 (+ 310us) server_negotiation.cc:407] Connection header received
0619 14:33:01.553248 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:33:01.553253 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:33:01.553308 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:33:01.553386 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:33:01.553826 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:33:01.554469 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:33:01.555560 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:33:01.555989 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:33:01.556109 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:33:01.556185 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 14:33:01.556354 (+ 169us) server_negotiation.cc:299] Negotiation successful
0619 14:33:01.556423 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 14:33:01.556576 12798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:33:01.552565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.3:43275 (local address 127.2.2.1:54849)
0619 14:33:01.552999 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0619 14:33:01.553011 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:33:01.553138 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:33:01.553404 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:33:01.553407 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:33:01.553425 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:33:01.553712 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:33:01.553723 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:33:01.554577 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:33:01.554582 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:33:01.555432 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:33:01.555441 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:33:01.556114 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:33:01.556118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:33:01.556218 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:33:01.556241 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 14:33:01.556340 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 14:33:01.556417 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":146,"threads_started":1}
I20260619 14:33:01.556589 12784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:33:01.552679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:51923 (local address 127.2.2.2:43235)
0619 14:33:01.552802 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 14:33:01.552808 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:33:01.553444 (+ 636us) server_negotiation.cc:407] Connection header received
0619 14:33:01.553588 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:33:01.553592 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:33:01.553639 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:33:01.553724 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:33:01.553977 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:33:01.554579 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:33:01.555607 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:33:01.556029 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:33:01.556148 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:33:01.556212 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 14:33:01.556394 (+ 182us) server_negotiation.cc:299] Negotiation successful
0619 14:33:01.556468 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 14:33:01.557160 12799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:33:01.552565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.2:43235 (local address 127.2.2.1:51923)
0619 14:33:01.553367 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0619 14:33:01.553382 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:33:01.553503 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:33:01.553734 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:33:01.553737 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:33:01.553754 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:33:01.553872 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:33:01.553877 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:33:01.554685 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:33:01.554691 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:33:01.555479 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:33:01.555487 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:33:01.556148 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:33:01.556152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:33:01.556259 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:33:01.556279 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 14:33:01.556379 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 14:33:01.556451 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"mutex_wait_us":28,"thread_start_us":119,"threads_started":1}
I20260619 14:33:01.557302 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7009ea6c22841f79575e30abbe3ab93" candidate_uuid: "d72375b43f114df5960175eaac61229d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be04ed08fcfc475eb48a39866616ac48" is_pre_election: true
I20260619 14:33:01.557682 12726 raft_consensus.cc:2454] T b7009ea6c22841f79575e30abbe3ab93 P be04ed08fcfc475eb48a39866616ac48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d72375b43f114df5960175eaac61229d in term 0.
I20260619 14:33:01.557839 12592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7009ea6c22841f79575e30abbe3ab93" candidate_uuid: "d72375b43f114df5960175eaac61229d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24036d7afbfa4100855591e1dd40f7af" is_pre_election: true
I20260619 14:33:01.558171 12592 raft_consensus.cc:2454] T b7009ea6c22841f79575e30abbe3ab93 P 24036d7afbfa4100855591e1dd40f7af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d72375b43f114df5960175eaac61229d in term 0.
I20260619 14:33:01.558279 12391 leader_election.cc:302] T b7009ea6c22841f79575e30abbe3ab93 P d72375b43f114df5960175eaac61229d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be04ed08fcfc475eb48a39866616ac48, d72375b43f114df5960175eaac61229d; no voters:
I20260619 14:33:01.558583 12796 raft_consensus.cc:2781] T b7009ea6c22841f79575e30abbe3ab93 P d72375b43f114df5960175eaac61229d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:33:01.558681 12796 raft_consensus.cc:482] T b7009ea6c22841f79575e30abbe3ab93 P d72375b43f114df5960175eaac61229d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:33:01.558737 12796 raft_consensus.cc:3037] T b7009ea6c22841f79575e30abbe3ab93 P d72375b43f114df5960175eaac61229d [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:33:01.559876 12796 raft_consensus.cc:504] T b7009ea6c22841f79575e30abbe3ab93 P d72375b43f114df5960175eaac61229d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24036d7afbfa4100855591e1dd40f7af" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 43235 } } peers { permanent_uuid: "be04ed08fcfc475eb48a39866616ac48" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 43275 } } peers { permanent_uuid: "d72375b43f114df5960175eaac61229d" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 36679 } }
I20260619 14:33:01.560256 12796 leader_election.cc:288] T b7009ea6c22841f79575e30abbe3ab93 P d72375b43f114df5960175eaac61229d [CANDIDATE]: Term 1 election: Requested vote from peers 24036d7afbfa4100855591e1dd40f7af (127.2.2.2:43235), be04ed08fcfc475eb48a39866616ac48 (127.2.2.3:43275)
I20260619 14:33:01.560583 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7009ea6c22841f79575e30abbe3ab93" candidate_uuid: "d72375b43f114df5960175eaac61229d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be04ed08fcfc475eb48a39866616ac48"
I20260619 14:33:01.560721 12592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7009ea6c22841f79575e30abbe3ab93" candidate_uuid: "d72375b43f114df5960175eaac61229d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24036d7afbfa4100855591e1dd40f7af"
I20260619 14:33:01.560750 12726 raft_consensus.cc:3037] T b7009ea6c22841f79575e30abbe3ab93 P be04ed08fcfc475eb48a39866616ac48 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:33:01.560843 12592 raft_consensus.cc:3037] T b7009ea6c22841f79575e30abbe3ab93 P 24036d7afbfa4100855591e1dd40f7af [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:33:01.562071 12726 raft_consensus.cc:2454] T b7009ea6c22841f79575e30abbe3ab93 P be04ed08fcfc475eb48a39866616ac48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d72375b43f114df5960175eaac61229d in term 1.
I20260619 14:33:01.562071 12592 raft_consensus.cc:2454] T b7009ea6c22841f79575e30abbe3ab93 P 24036d7afbfa4100855591e1dd40f7af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d72375b43f114df5960175eaac61229d in term 1.
I20260619 14:33:01.562386 12391 leader_election.cc:302] T b7009ea6c22841f79575e30abbe3ab93 P d72375b43f114df5960175eaac61229d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be04ed08fcfc475eb48a39866616ac48, d72375b43f114df5960175eaac61229d; no voters:
I20260619 14:33:01.562618 12796 raft_consensus.cc:2781] T b7009ea6c22841f79575e30abbe3ab93 P d72375b43f114df5960175eaac61229d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:33:01.562844 12796 raft_consensus.cc:686] T b7009ea6c22841f79575e30abbe3ab93 P d72375b43f114df5960175eaac61229d [term 1 LEADER]: Becoming Leader. State: Replica: d72375b43f114df5960175eaac61229d, State: Running, Role: LEADER
I20260619 14:33:01.563171 12796 consensus_queue.cc:237] T b7009ea6c22841f79575e30abbe3ab93 P d72375b43f114df5960175eaac61229d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24036d7afbfa4100855591e1dd40f7af" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 43235 } } peers { permanent_uuid: "be04ed08fcfc475eb48a39866616ac48" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 43275 } } peers { permanent_uuid: "d72375b43f114df5960175eaac61229d" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 36679 } }
I20260619 14:33:01.566787 12313 catalog_manager.cc:5495] T b7009ea6c22841f79575e30abbe3ab93 P d72375b43f114df5960175eaac61229d reported cstate change: term changed from 0 to 1, leader changed from <none> to d72375b43f114df5960175eaac61229d (127.2.2.1). New cstate: current_term: 1 leader_uuid: "d72375b43f114df5960175eaac61229d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24036d7afbfa4100855591e1dd40f7af" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 43235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be04ed08fcfc475eb48a39866616ac48" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 43275 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d72375b43f114df5960175eaac61229d" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 36679 } health_report { overall_health: HEALTHY } } }
I20260619 14:33:01.587044 12787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:33:01.581992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50978 (local address 127.2.2.1:36679)
0619 14:33:01.582233 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0619 14:33:01.582238 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:33:01.582261 (+ 23us) server_negotiation.cc:407] Connection header received
0619 14:33:01.582395 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:33:01.582401 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:33:01.582496 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:33:01.582639 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:33:01.582990 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:33:01.583978 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:33:01.585086 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:33:01.585437 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:33:01.585571 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:33:01.585917 (+ 346us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:33:01.586579 (+ 662us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:33:01.586724 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 14:33:01.586815 (+ 91us) server_negotiation.cc:299] Negotiation successful
0619 14:33:01.586882 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 14:33:01.593448 12726 raft_consensus.cc:1261] T b7009ea6c22841f79575e30abbe3ab93 P be04ed08fcfc475eb48a39866616ac48 [term 1 FOLLOWER]: Refusing update from remote peer d72375b43f114df5960175eaac61229d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:33:01.593562 12592 raft_consensus.cc:1261] T b7009ea6c22841f79575e30abbe3ab93 P 24036d7afbfa4100855591e1dd40f7af [term 1 FOLLOWER]: Refusing update from remote peer d72375b43f114df5960175eaac61229d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:33:01.594309 12800 consensus_queue.cc:1035] T b7009ea6c22841f79575e30abbe3ab93 P d72375b43f114df5960175eaac61229d [LEADER]: Connected to new peer: Peer: permanent_uuid: "be04ed08fcfc475eb48a39866616ac48" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 43275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:33:01.594587 12796 consensus_queue.cc:1035] T b7009ea6c22841f79575e30abbe3ab93 P d72375b43f114df5960175eaac61229d [LEADER]: Connected to new peer: Peer: permanent_uuid: "24036d7afbfa4100855591e1dd40f7af" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 43235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:33:01.603076 12805 mvcc.cc:205] Tried to move back new op lower bound from 7298578766199013376 to 7298578766086090752. Current Snapshot: MvccSnapshot[applied={T|T < 7298578766199013376}]
I20260619 14:33:01.615165 12313 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:33:01.615336 12313 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:33:01.617874 12313 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 177424109442400da960fcff8604a0bb: Sending DeleteTablet for 3 replicas of tablet b7009ea6c22841f79575e30abbe3ab93
I20260619 14:33:01.618775 12572 tablet_service.cc:1514] Processing DeleteTablet for tablet b7009ea6c22841f79575e30abbe3ab93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:33:01 UTC) from {username='kudu', principal='kudu/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:39252
I20260619 14:33:01.618902 12438 tablet_service.cc:1514] Processing DeleteTablet for tablet b7009ea6c22841f79575e30abbe3ab93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:33:01 UTC) from {username='kudu', principal='kudu/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:50976
I20260619 14:33:01.618949 12706 tablet_service.cc:1514] Processing DeleteTablet for tablet b7009ea6c22841f79575e30abbe3ab93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:33:01 UTC) from {username='kudu', principal='kudu/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:53374
I20260619 14:33:01.619354 12815 tablet_replica.cc:330] T b7009ea6c22841f79575e30abbe3ab93 P d72375b43f114df5960175eaac61229d: stopping tablet replica
I20260619 14:33:01.619696 12815 raft_consensus.cc:2229] T b7009ea6c22841f79575e30abbe3ab93 P d72375b43f114df5960175eaac61229d [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:33:01.619763 12816 tablet_replica.cc:330] T b7009ea6c22841f79575e30abbe3ab93 P be04ed08fcfc475eb48a39866616ac48: stopping tablet replica
I20260619 14:33:01.620138 12815 raft_consensus.cc:2258] T b7009ea6c22841f79575e30abbe3ab93 P d72375b43f114df5960175eaac61229d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:33:01.620273 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 12375
I20260619 14:33:01.620278 12816 raft_consensus.cc:2229] T b7009ea6c22841f79575e30abbe3ab93 P be04ed08fcfc475eb48a39866616ac48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:33:01.620520 12814 tablet_replica.cc:330] T b7009ea6c22841f79575e30abbe3ab93 P 24036d7afbfa4100855591e1dd40f7af: stopping tablet replica
I20260619 14:33:01.620923 12814 raft_consensus.cc:2229] T b7009ea6c22841f79575e30abbe3ab93 P 24036d7afbfa4100855591e1dd40f7af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:33:01.621227 12814 raft_consensus.cc:2258] T b7009ea6c22841f79575e30abbe3ab93 P 24036d7afbfa4100855591e1dd40f7af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:33:01.620911 12816 raft_consensus.cc:2258] T b7009ea6c22841f79575e30abbe3ab93 P be04ed08fcfc475eb48a39866616ac48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:33:01.622855 12814 ts_tablet_manager.cc:1905] T b7009ea6c22841f79575e30abbe3ab93 P 24036d7afbfa4100855591e1dd40f7af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:33:01.622871 12816 ts_tablet_manager.cc:1905] T b7009ea6c22841f79575e30abbe3ab93 P be04ed08fcfc475eb48a39866616ac48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:33:01.625108 12814 ts_tablet_manager.cc:1918] T b7009ea6c22841f79575e30abbe3ab93 P 24036d7afbfa4100855591e1dd40f7af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:33:01.625228 12814 log.cc:1196] T b7009ea6c22841f79575e30abbe3ab93 P 24036d7afbfa4100855591e1dd40f7af: Deleting WAL directory at /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/ts-1/wal/wals/b7009ea6c22841f79575e30abbe3ab93
I20260619 14:33:01.625528 12814 ts_tablet_manager.cc:1939] T b7009ea6c22841f79575e30abbe3ab93 P 24036d7afbfa4100855591e1dd40f7af: Deleting consensus metadata
I20260619 14:33:01.626273 12816 ts_tablet_manager.cc:1918] T b7009ea6c22841f79575e30abbe3ab93 P be04ed08fcfc475eb48a39866616ac48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:33:01.626389 12816 log.cc:1196] T b7009ea6c22841f79575e30abbe3ab93 P be04ed08fcfc475eb48a39866616ac48: Deleting WAL directory at /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879514993456-2056-0/minicluster-data/ts-2/wal/wals/b7009ea6c22841f79575e30abbe3ab93
I20260619 14:33:01.626726 12816 ts_tablet_manager.cc:1939] T b7009ea6c22841f79575e30abbe3ab93 P be04ed08fcfc475eb48a39866616ac48: Deleting consensus metadata
I20260619 14:33:01.626899 12300 catalog_manager.cc:4841] TS 24036d7afbfa4100855591e1dd40f7af (127.2.2.2:43235): tablet b7009ea6c22841f79575e30abbe3ab93 (table test-table [id=d6731a7b60a040208a1c79cdbbd491b9]) successfully deleted
I20260619 14:33:01.627959 12297 catalog_manager.cc:4841] TS be04ed08fcfc475eb48a39866616ac48 (127.2.2.3:43275): tablet b7009ea6c22841f79575e30abbe3ab93 (table test-table [id=d6731a7b60a040208a1c79cdbbd491b9]) successfully deleted
W20260619 14:33:01.628571 12299 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.2.1:36679 (error 108)
I20260619 14:33:01.628719 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 12509
I20260619 14:33:01.630163 12783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:33:01.629851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.1:36679 (local address 127.0.0.1:50984)
0619 14:33:01.629962 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 14:33:01.630032 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.1:36679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260619 14:33:01.630347 12299 catalog_manager.cc:4584] TS d72375b43f114df5960175eaac61229d (127.2.2.1:36679): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b7009ea6c22841f79575e30abbe3ab93: Network error: Client connection negotiation failed: client connection to 127.2.2.1:36679: connect: Connection refused (error 111)
I20260619 14:33:01.634454 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 12643
I20260619 14:33:01.640625 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 12281
2026-06-19T14:33:01Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2341 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:33:01 dist-test-slave-70rm krb5kdc[12820](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:33:01 dist-test-slave-70rm krb5kdc[12820](info): set up 2 sockets
Jun 19 14:33:01 dist-test-slave-70rm krb5kdc[12820](info): commencing operation
krb5kdc: starting...
W20260619 14:33:02.687638 2056 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Jun 19 14:33:02 dist-test-slave-70rm krb5kdc[12820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:33:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:33:02Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.2.2.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.2.2.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.2.2.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/krb5kdc/rangeradmin_127.2.2.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.2.2.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/krb5kdc/rangeradmin_127.2.2.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.2.2.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.2.2.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.2.2.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/krb5kdc/rangerlookup_127.2.2.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.2.2.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/krb5kdc/rangerlookup_127.2.2.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.2.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.2.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.2.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/krb5kdc/HTTP_127.2.2.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.2.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/krb5kdc/HTTP_127.2.2.20@KRBTEST.COM.keytab.
I20260619 14:33:02.861661 2056 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskMyx_Zw/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/postgres -l logfile start
2026-06-19 14:33:05.340 UTC [12850] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 14:33:05.341 UTC [12850] LOG: listening on IPv4 address "127.2.2.20", port 40177
2026-06-19 14:33:05.344 UTC [12850] LOG: listening on Unix socket "/tmp/.s.PGSQL.40177"
2026-06-19 14:33:05.367 UTC [12853] LOG: database system was shut down at 2026-06-19 14:33:03 UTC
2026-06-19 14:33:05.371 UTC [12850] LOG: database system is ready to accept connections
I20260619 14:33:06.326860 2056 mini_postgres.cc:96] Postgres bound to 40177
2026-06-19 14:33:06.333 UTC [12861] FATAL: database "slave" does not exist
127.2.2.20:40177 - accepting connections
I20260619 14:33:06.333844 2056 mini_ranger.cc:161] Starting Ranger...
I20260619 14:33:06.349006 2056 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 14:33:06.837625 2056 mini_ranger.cc:87] Created ranger Postgres database
I20260619 14:33:06.837723 2056 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ranger-admin
2026-06-19 14:33:07,121 [I] DB FLAVOR :POSTGRES
2026-06-19 14:33:07,121 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 14:33:07,121 [I] Checking connection..
2026-06-19 14:33:07,121 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMyx_Zw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMyx_Zw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.2.20:40177/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 14:33:07,376 [I] Checking connection passed.
2026-06-19 14:33:07,377 [I] --------- Verifying version history table ---------
2026-06-19 14:33:07,377 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMyx_Zw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMyx_Zw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.2.20:40177/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 14:33:07,640 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 14:33:07,641 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMyx_Zw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMyx_Zw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.2.20:40177/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 14:33:07,871 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 14:33:07,872 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 14:33:07,872 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMyx_Zw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMyx_Zw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.2.20:40177/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskMyx_Zw/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 14:33:08.101 UTC [12925] WARNING: there is no transaction in progress
2026-06-19 14:33:08,107 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 14:33:08,108 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMyx_Zw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMyx_Zw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.2.20:40177/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 14:33:08,368 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 14:33:08,369 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 14:33:08,369 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMyx_Zw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMyx_Zw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.2.20:40177/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 14:33:08,597 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMyx_Zw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMyx_Zw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.2.20:40177/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 14:33:08,840 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMyx_Zw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMyx_Zw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.2.20:40177/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-70rm.c.gcp-upstream.internal','N') ;"
2026-06-19 14:33:09,080 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMyx_Zw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMyx_Zw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.2.20:40177/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 14:33:09,347 [I] Table x_portal_user does not exist in database ranger
2026-06-19 14:33:09,347 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMyx_Zw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMyx_Zw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.2.20:40177/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 14:33:09,583 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 14:33:09,584 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMyx_Zw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMyx_Zw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.2.20:40177/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 14:33:09,827 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMyx_Zw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMyx_Zw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.2.20:40177/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 14:33:10,074 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 14:33:10,074 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMyx_Zw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMyx_Zw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.2.20:40177/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskMyx_Zw/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 14:33:11.097 UTC [13033] WARNING: there is no transaction in progress
2026-06-19 14:33:11.115 UTC [13033] WARNING: there is no transaction in progress
2026-06-19 14:33:11.136 UTC [13033] WARNING: there is no transaction in progress
2026-06-19 14:33:11.154 UTC [13033] WARNING: there is no transaction in progress
2026-06-19 14:33:11.176 UTC [13033] WARNING: there is no transaction in progress
2026-06-19 14:33:11.197 UTC [13033] WARNING: there is no transaction in progress
2026-06-19 14:33:11.258 UTC [13033] WARNING: there is no transaction in progress
2026-06-19 14:33:11.266 UTC [13033] WARNING: there is no transaction in progress
2026-06-19 14:33:11.283 UTC [13033] WARNING: there is no transaction in progress
2026-06-19 14:33:11.294 UTC [13033] WARNING: there is no transaction in progress
2026-06-19 14:33:11.306 UTC [13033] WARNING: there is no transaction in progress
2026-06-19 14:33:11.322 UTC [13033] WARNING: there is no transaction in progress
2026-06-19 14:33:11.332 UTC [13033] WARNING: there is no transaction in progress
2026-06-19 14:33:11.339 UTC [13033] WARNING: there is no transaction in progress
2026-06-19 14:33:11.346 UTC [13033] WARNING: there is no transaction in progress
2026-06-19 14:33:11.353 UTC [13033] WARNING: there is no transaction in progress
2026-06-19 14:33:12,069 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 14:33:12,070 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMyx_Zw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMyx_Zw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.2.20:40177/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 14:33:12,315 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 14:33:12,315 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMyx_Zw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMyx_Zw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.2.20:40177/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 14:33:12,552 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 14:33:12,552 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMyx_Zw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMyx_Zw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.2.20:40177/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 14:33:12,792 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 14:33:12,792 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMyx_Zw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMyx_Zw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.2.20:40177/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 14:33:13,030 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMyx_Zw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMyx_Zw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.2.20:40177/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 14:33:13,266 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMyx_Zw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMyx_Zw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.2.20:40177/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-70rm.c.gcp-upstream.internal';"
2026-06-19 14:33:13,536 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 14:33:13,536 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMyx_Zw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMyx_Zw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.2.20:40177/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 14:33:13,786 [I] DB_PATCHES have already been applied
I20260619 14:33:13.792176 2056 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ranger-admin:/tmp/dist-test-taskMyx_Zw/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskMyx_Zw/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskMyx_Zw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMyx_Zw/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 14:33:13.792249 2056 mini_ranger.cc:193] Using host: 127.2.2.20
I20260619 14:33:13.796028 2056 mini_ranger.cc:227] Ranger admin URL: http://127.2.2.20:56903
Jun 19, 2026 2:33:14 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 2:33:14 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 2:33:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 2:33:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 2:33:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 2:33:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 2:33:14 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 14:33:14.933960 2056 mini_ranger.cc:160] Time spent starting Ranger: real 8.600s user 0.001s sys 0.010s
Jun 19, 2026 2:33:34 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 2:33:34 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Jun 19, 2026 2:33:34 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 2:33:34 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 2:33:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 2:33:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 2:33:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 2:33:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 2:33:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 2:33:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 2:33:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 2:33:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 2:33:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 2:33:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 2:33:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 2:33:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 2:33:34 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 2:33:35 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 14:33:36.310231 2056 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.2.2.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.2.2.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.2.2.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/krb5kdc/rangerkms_127.2.2.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.2.2.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/krb5kdc/rangerkms_127.2.2.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.2.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.2.2.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.2.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/krb5kdc/HTTP_127.2.2.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.2.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/krb5kdc/HTTP_127.2.2.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 14:33:36 dist-test-slave-70rm krb5kdc[12820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879616, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 14:33:36.400265 2056 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 14:33:36.410529 2056 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 14:33:37.260957 2056 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 14:33:37.261053 2056 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ranger-kms
I20260619 14:33:37.261152 2056 mini_ranger_kms.cc:227] Using postgres at 127.2.2.20:40177
2026-06-19 14:33:37,458 [I] DB FLAVOR :POSTGRES
2026-06-19 14:33:37,459 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 14:33:37,459 [I] Checking connection
2026-06-19 14:33:37,459 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMyx_Zw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.2.20:40177/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 14:33:37,704 [I] connection success
2026-06-19 14:33:37,704 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 14:33:37,704 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 14:33:37,704 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMyx_Zw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.2.20:40177/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 14:33:37,953 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 14:33:37,954 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 14:33:37,954 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 14:33:37,954 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMyx_Zw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.2.20:40177/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 14:33:38,208 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 14:33:38.677038 2056 mini_ranger_kms.cc:310] Created kms service
I20260619 14:33:38.903879 2056 mini_ranger_kms.cc:326] Created kudu user
I20260619 14:33:38.949949 2056 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 14:33:38 dist-test-slave-70rm krb5kdc[12820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879616, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.2.2.20@KRBTEST.COM
I20260619 14:33:39.334081 2056 mini_ranger_kms.cc:384] Added ranger policy
I20260619 14:33:39.334281 2056 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ranger-kms:/tmp/dist-test-taskMyx_Zw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMyx_Zw/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskMyx_Zw/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 14:33:39.334304 2056 mini_ranger_kms.cc:240] Using host: 127.2.2.20
I20260619 14:33:39.338424 2056 mini_ranger_kms.cc:276] Ranger KMS PID: 13213
I20260619 14:33:39.338574 2056 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.2.2.20:43581
Jun 19, 2026 2:33:40 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 2:33:40 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 2:33:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 2:33:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 2:33:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 2:33:40 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 14:33:41.558825 2056 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.159s user 0.011s sys 0.171s
I20260619 14:33:41.558993 2056 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 14:33:41.559113 2056 mini_ranger_kms.cc:401] 127.2.2.20:43581/kms/v1/keys
Jun 19 14:33:41 dist-test-slave-70rm krb5kdc[12820](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781879621, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.2.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 14:33:42 dist-test-slave-70rm krb5kdc[12820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879621, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.2.20@KRBTEST.COM for HTTP/127.2.2.20@KRBTEST.COM
Jun 19 14:33:42 dist-test-slave-70rm krb5kdc[12820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879621, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.2.20@KRBTEST.COM for HTTP/127.2.2.20@KRBTEST.COM
Jun 19 14:33:44 dist-test-slave-70rm krb5kdc[12820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.2.2.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab.
Entry for principal kudu/127.2.2.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab.
I20260619 14:33:44.885440 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:42665
--webserver_interface=127.2.2.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:37049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.2.62:42665
--ranger_config_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.2.20:43581/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:33:45.000198 13264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:33:45.000536 13264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:33:45.000609 13264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:33:45.004761 13264 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:33:45.004873 13264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:33:45.004909 13264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:33:45.004946 13264 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:33:45.004985 13264 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:33:45.010892 13264 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:37049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/master-0/wal
--ranger_kms_url=127.2.2.20:43581/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.2.62:42665
--ranger_config_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:42665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.2.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13264
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:33:45.012102 13264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:33:45.013173 13264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:33:45.022719 13270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:33:45.022959 13264 server_base.cc:1034] running on GCE node
W20260619 14:33:45.023232 13269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:33:45.023496 13272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:33:45.024181 13264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:33:45.025647 13264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:33:45.026860 13264 hybrid_clock.cc:648] HybridClock initialized: now 1781879625026829 us; error 43 us; skew 500 ppm
Jun 19 14:33:45 dist-test-slave-70rm krb5kdc[12820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879625, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:33:45.030387 13264 init.cc:377] Logged in from keytab as kudu/127.2.2.62@KRBTEST.COM (short username kudu)
I20260619 14:33:45.031518 13264 webserver.cc:460] Webserver started at http://127.2.2.62:36389/ using document root <none> and password file <none>
I20260619 14:33:45.032156 13264 fs_manager.cc:359] Metadata directory not provided
I20260619 14:33:45.032224 13264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:33:45.032456 13264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 14:33:45 dist-test-slave-70rm krb5kdc[12820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879625, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.62@KRBTEST.COM for HTTP/127.2.2.20@KRBTEST.COM
I20260619 14:33:45.081326 13264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/master-0/data/instance:
uuid: "1c7f8dcef5ec43b6b5057c86646d62eb"
format_stamp: "Formatted at 2026-06-19 14:33:45 on dist-test-slave-70rm"
server_key: "b87a6db0e8d4142f2f5164063f34afb8"
server_key_iv: "e2e6c5501b12289b7643d0b67364a016"
server_key_version: "kuduclusterkey@0"
I20260619 14:33:45.081976 13264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/master-0/wal/instance:
uuid: "1c7f8dcef5ec43b6b5057c86646d62eb"
format_stamp: "Formatted at 2026-06-19 14:33:45 on dist-test-slave-70rm"
server_key: "b87a6db0e8d4142f2f5164063f34afb8"
server_key_iv: "e2e6c5501b12289b7643d0b67364a016"
server_key_version: "kuduclusterkey@0"
I20260619 14:33:45.086331 13264 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260619 14:33:45.100399 13279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:33:45.101912 13264 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:33:45.102059 13264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/master-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/master-0/wal
uuid: "1c7f8dcef5ec43b6b5057c86646d62eb"
format_stamp: "Formatted at 2026-06-19 14:33:45 on dist-test-slave-70rm"
server_key: "b87a6db0e8d4142f2f5164063f34afb8"
server_key_iv: "e2e6c5501b12289b7643d0b67364a016"
server_key_version: "kuduclusterkey@0"
I20260619 14:33:45.102243 13264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:33:45.121482 13264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:33:45.124688 13264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:33:45.124969 13264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:33:45.133503 13264 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.62:42665
I20260619 14:33:45.133515 13331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.62:42665 every 8 connection(s)
I20260619 14:33:45.134644 13264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
I20260619 14:33:45.138265 13332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:33:45.143473 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 13264
I20260619 14:33:45.143636 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/master-0/wal/instance
I20260619 14:33:45.143915 2056 external_mini_cluster.cc:1428] Setting key 9250479ac2fe3e05057b4e2c151e8592
I20260619 14:33:45.147267 13332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c7f8dcef5ec43b6b5057c86646d62eb: Bootstrap starting.
I20260619 14:33:45.150262 13332 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c7f8dcef5ec43b6b5057c86646d62eb: Neither blocks nor log segments found. Creating new log.
I20260619 14:33:45.151319 13332 log.cc:824] T 00000000000000000000000000000000 P 1c7f8dcef5ec43b6b5057c86646d62eb: Log is configured to *not* fsync() on all Append() calls
Jun 19 14:33:45 dist-test-slave-70rm krb5kdc[12820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:33:45.153682 13332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c7f8dcef5ec43b6b5057c86646d62eb: No bootstrap required, opened a new log
I20260619 14:33:45.158319 13335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:33:45.145451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52578 (local address 127.2.2.62:42665)
0619 14:33:45.145931 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0619 14:33:45.145946 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:33:45.145983 (+ 37us) server_negotiation.cc:407] Connection header received
0619 14:33:45.146745 (+ 762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:33:45.146764 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:33:45.147144 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:33:45.147707 (+ 563us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:33:45.148385 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:33:45.149409 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:33:45.150369 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:33:45.150858 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:33:45.151040 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:33:45.153710 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:33:45.153715 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:33:45.153729 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:33:45.153777 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:33:45.155489 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:33:45.155935 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:33:45.155940 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:33:45.155946 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:33:45.156056 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:33:45.156340 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:33:45.156345 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:33:45.156346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:33:45.156753 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:33:45.156963 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0619 14:33:45.157197 (+ 234us) server_negotiation.cc:299] Negotiation successful
0619 14:33:45.157491 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":159,"threads_started":1}
I20260619 14:33:45.158315 13332 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c7f8dcef5ec43b6b5057c86646d62eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c7f8dcef5ec43b6b5057c86646d62eb" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 42665 } }
I20260619 14:33:45.159682 13332 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c7f8dcef5ec43b6b5057c86646d62eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:33:45.159795 13332 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c7f8dcef5ec43b6b5057c86646d62eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c7f8dcef5ec43b6b5057c86646d62eb, State: Initialized, Role: FOLLOWER
I20260619 14:33:45.160310 13332 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c7f8dcef5ec43b6b5057c86646d62eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c7f8dcef5ec43b6b5057c86646d62eb" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 42665 } }
I20260619 14:33:45.160471 13332 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c7f8dcef5ec43b6b5057c86646d62eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:33:45.160563 13332 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c7f8dcef5ec43b6b5057c86646d62eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:33:45.160670 13332 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c7f8dcef5ec43b6b5057c86646d62eb [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:33:45.162254 13332 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c7f8dcef5ec43b6b5057c86646d62eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c7f8dcef5ec43b6b5057c86646d62eb" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 42665 } }
I20260619 14:33:45.162623 13332 leader_election.cc:302] T 00000000000000000000000000000000 P 1c7f8dcef5ec43b6b5057c86646d62eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c7f8dcef5ec43b6b5057c86646d62eb; no voters:
I20260619 14:33:45.162973 13332 leader_election.cc:288] T 00000000000000000000000000000000 P 1c7f8dcef5ec43b6b5057c86646d62eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:33:45.163087 13337 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c7f8dcef5ec43b6b5057c86646d62eb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:33:45.163367 13337 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c7f8dcef5ec43b6b5057c86646d62eb [term 1 LEADER]: Becoming Leader. State: Replica: 1c7f8dcef5ec43b6b5057c86646d62eb, State: Running, Role: LEADER
I20260619 14:33:45.163722 13337 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c7f8dcef5ec43b6b5057c86646d62eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c7f8dcef5ec43b6b5057c86646d62eb" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 42665 } }
I20260619 14:33:45.164242 13332 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c7f8dcef5ec43b6b5057c86646d62eb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:33:45.164840 13337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c7f8dcef5ec43b6b5057c86646d62eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c7f8dcef5ec43b6b5057c86646d62eb. Latest consensus state: current_term: 1 leader_uuid: "1c7f8dcef5ec43b6b5057c86646d62eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c7f8dcef5ec43b6b5057c86646d62eb" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 42665 } } }
I20260619 14:33:45.164952 13337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c7f8dcef5ec43b6b5057c86646d62eb [sys.catalog]: This master's current role is: LEADER
I20260619 14:33:45.165295 13339 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c7f8dcef5ec43b6b5057c86646d62eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c7f8dcef5ec43b6b5057c86646d62eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c7f8dcef5ec43b6b5057c86646d62eb" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 42665 } } }
I20260619 14:33:45.166416 13339 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c7f8dcef5ec43b6b5057c86646d62eb [sys.catalog]: This master's current role is: LEADER
I20260619 14:33:45.167203 13343 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:33:45.167222 13332 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 14:33:45.172878 13343 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:33:45.181488 13343 catalog_manager.cc:1347] Generated new cluster ID: ba2e875d9b0d44e395d9e7a923b7f28e
I20260619 14:33:45.181577 13343 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:33:45.207352 13343 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:33:45.208724 13343 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:33:45.213762 13343 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c7f8dcef5ec43b6b5057c86646d62eb: Generated new TSK 0
I20260619 14:33:45.214636 13343 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.2.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab.
Entry for principal kudu/127.2.2.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab.
I20260619 14:33:45.881124 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.2.20:43581/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.1:0
--local_ip_for_outbound_sockets=127.2.2.1
--webserver_interface=127.2.2.1
--webserver_port=0
--tserver_master_addrs=127.2.2.62:42665
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab
--principal=kudu/127.2.2.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:37049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:33:46.033608 13377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:33:46.033917 13377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:33:46.034015 13377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:33:46.038424 13377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:33:46.038551 13377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:33:46.038664 13377 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.1
I20260619 14:33:46.044365 13377 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:37049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.2.2.20:43581/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab
--principal=kudu/127.2.2.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.2.1
--webserver_port=0
--tserver_master_addrs=127.2.2.62:42665
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13377
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.1
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:33:46.045542 13377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:33:46.046885 13377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:33:46.057708 13383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:33:46.057708 13382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:33:46.058220 13385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:33:46.058358 13377 server_base.cc:1034] running on GCE node
I20260619 14:33:46.058874 13377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:33:46.059544 13377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:33:46.060750 13377 hybrid_clock.cc:648] HybridClock initialized: now 1781879626060716 us; error 49 us; skew 500 ppm
Jun 19 14:33:46 dist-test-slave-70rm krb5kdc[12820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879626, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:33:46.063832 13377 init.cc:377] Logged in from keytab as kudu/127.2.2.1@KRBTEST.COM (short username kudu)
I20260619 14:33:46.064841 13377 webserver.cc:460] Webserver started at http://127.2.2.1:40273/ using document root <none> and password file <none>
I20260619 14:33:46.065505 13377 fs_manager.cc:359] Metadata directory not provided
I20260619 14:33:46.065594 13377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:33:46.065814 13377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 14:33:46 dist-test-slave-70rm krb5kdc[12820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879626, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.1@KRBTEST.COM for HTTP/127.2.2.20@KRBTEST.COM
I20260619 14:33:46.082472 13377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ts-0/data/instance:
uuid: "eef8bc152baa4506bf365e3f76304b75"
format_stamp: "Formatted at 2026-06-19 14:33:46 on dist-test-slave-70rm"
server_key: "9a34fc5d49ee5d256dbba1eeba3cb769"
server_key_iv: "efd15ef485e625b2a404ca24869be4a8"
server_key_version: "kuduclusterkey@0"
I20260619 14:33:46.083379 13377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ts-0/wal/instance:
uuid: "eef8bc152baa4506bf365e3f76304b75"
format_stamp: "Formatted at 2026-06-19 14:33:46 on dist-test-slave-70rm"
server_key: "9a34fc5d49ee5d256dbba1eeba3cb769"
server_key_iv: "efd15ef485e625b2a404ca24869be4a8"
server_key_version: "kuduclusterkey@0"
I20260619 14:33:46.096542 13377 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.001s sys 0.002s
I20260619 14:33:46.138904 13394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:33:46.140790 13377 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 14:33:46.140959 13377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ts-0/wal
uuid: "eef8bc152baa4506bf365e3f76304b75"
format_stamp: "Formatted at 2026-06-19 14:33:46 on dist-test-slave-70rm"
server_key: "9a34fc5d49ee5d256dbba1eeba3cb769"
server_key_iv: "efd15ef485e625b2a404ca24869be4a8"
server_key_version: "kuduclusterkey@0"
I20260619 14:33:46.141134 13377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:33:46.167040 13377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:33:46.174649 13377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:33:46.174959 13377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:33:46.175724 13377 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:33:46.176841 13377 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:33:46.176944 13377 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:33:46.177062 13377 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:33:46.177140 13377 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:33:46.191280 13377 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.1:36723
I20260619 14:33:46.192632 13377 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
I20260619 14:33:46.194552 13507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.1:36723 every 8 connection(s)
I20260619 14:33:46.200685 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 13377
I20260619 14:33:46.200822 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ts-0/wal/instance
I20260619 14:33:46.201117 2056 external_mini_cluster.cc:1428] Setting key b01ed67763c4770f47918bc490169d43
Jun 19 14:33:46 dist-test-slave-70rm krb5kdc[12820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879626, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.1@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:33:46.225244 13510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:33:46.197311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:42665 (local address 127.2.2.1:56731)
0619 14:33:46.197991 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0619 14:33:46.198056 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0619 14:33:46.199154 (+ 1098us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:33:46.200041 (+ 887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:33:46.200051 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:33:46.200423 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:33:46.201138 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:33:46.201158 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:33:46.206283 (+ 5125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:33:46.206288 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:33:46.207872 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:33:46.207889 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:33:46.208707 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:33:46.208780 (+ 73us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:33:46.209005 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:33:46.209741 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:33:46.209765 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:33:46.211527 (+ 1762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:33:46.221461 (+ 9934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:33:46.221472 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:33:46.221495 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:33:46.221945 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:33:46.222353 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:33:46.222357 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:33:46.222359 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:33:46.222570 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:33:46.223034 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:33:46.223042 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:33:46.223459 (+ 417us) client_negotiation.cc:769] Sending connection context
0619 14:33:46.223708 (+ 249us) client_negotiation.cc:240] Negotiation successful
0619 14:33:46.224118 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":172,"threads_started":1}
I20260619 14:33:46.226372 13511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:33:46.198566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:56731 (local address 127.2.2.62:42665)
0619 14:33:46.199597 (+ 1031us) server_negotiation.cc:206] Beginning negotiation
0619 14:33:46.199605 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:33:46.199620 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:33:46.199664 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:33:46.199667 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:33:46.199714 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:33:46.201304 (+ 1590us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:33:46.201410 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:33:46.202183 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:33:46.208068 (+ 5885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:33:46.208453 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:33:46.208557 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:33:46.211753 (+ 3196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:33:46.211757 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:33:46.211760 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:33:46.211789 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:33:46.221283 (+ 9494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:33:46.222096 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:33:46.222100 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:33:46.222102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:33:46.222231 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:33:46.222686 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:33:46.222689 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:33:46.222690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:33:46.222915 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:33:46.223557 (+ 642us) server_negotiation.cc:1035] Waiting for connection context
0619 14:33:46.226014 (+ 2457us) server_negotiation.cc:299] Negotiation successful
0619 14:33:46.226189 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":948,"thread_start_us":123,"threads_started":1}
I20260619 14:33:46.227445 13508 heartbeater.cc:347] Connected to a master server at 127.2.2.62:42665
I20260619 14:33:46.227710 13508 heartbeater.cc:464] Registering TS with master...
I20260619 14:33:46.228314 13508 heartbeater.cc:511] Master 127.2.2.62:42665 requested a full tablet report, sending...
I20260619 14:33:46.230018 13296 ts_manager.cc:194] Registered new tserver with Master: eef8bc152baa4506bf365e3f76304b75 (127.2.2.1:36723)
I20260619 14:33:46.231645 13296 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.2.1@KRBTEST.COM'} at 127.2.2.1:56731
WARNING: no policy specified for kudu/127.2.2.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab.
Entry for principal kudu/127.2.2.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab.
I20260619 14:33:46.273692 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.2.20:43581/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.2:0
--local_ip_for_outbound_sockets=127.2.2.2
--webserver_interface=127.2.2.2
--webserver_port=0
--tserver_master_addrs=127.2.2.62:42665
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab
--principal=kudu/127.2.2.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:37049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:33:46 dist-test-slave-70rm krb5kdc[12820](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781879626, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 14:33:46.436554 13516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:33:46.436924 13516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:33:46.436990 13516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:33:46.443001 13516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:33:46.443090 13516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:33:46.443181 13516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.2
I20260619 14:33:46.451220 13516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:37049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.2.2.20:43581/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab
--principal=kudu/127.2.2.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.2.2
--webserver_port=0
--tserver_master_addrs=127.2.2.62:42665
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.2
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:33:46.452643 13516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:33:46.453990 13516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:33:46.466486 13525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:33:46.466719 13522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:33:46.466521 13523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:33:46.468029 13516 server_base.cc:1034] running on GCE node
I20260619 14:33:46.468580 13516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:33:46.469436 13516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:33:46.476886 13516 hybrid_clock.cc:648] HybridClock initialized: now 1781879626476835 us; error 46 us; skew 500 ppm
Jun 19 14:33:46 dist-test-slave-70rm krb5kdc[12820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879626, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:33:46.480999 13516 init.cc:377] Logged in from keytab as kudu/127.2.2.2@KRBTEST.COM (short username kudu)
I20260619 14:33:46.482372 13516 webserver.cc:460] Webserver started at http://127.2.2.2:44597/ using document root <none> and password file <none>
I20260619 14:33:46.483114 13516 fs_manager.cc:359] Metadata directory not provided
I20260619 14:33:46.483206 13516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:33:46.483470 13516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 14:33:46 dist-test-slave-70rm krb5kdc[12820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879626, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.2@KRBTEST.COM for HTTP/127.2.2.20@KRBTEST.COM
I20260619 14:33:46.500888 13516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ts-1/data/instance:
uuid: "ff1fc2db7def4d15a7cc82e9f6ab7eb4"
format_stamp: "Formatted at 2026-06-19 14:33:46 on dist-test-slave-70rm"
server_key: "147066d3877272c23114958578060a3b"
server_key_iv: "a9eec70c2b59d934aacb64a08989d2c1"
server_key_version: "kuduclusterkey@0"
I20260619 14:33:46.501549 13516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ts-1/wal/instance:
uuid: "ff1fc2db7def4d15a7cc82e9f6ab7eb4"
format_stamp: "Formatted at 2026-06-19 14:33:46 on dist-test-slave-70rm"
server_key: "147066d3877272c23114958578060a3b"
server_key_iv: "a9eec70c2b59d934aacb64a08989d2c1"
server_key_version: "kuduclusterkey@0"
I20260619 14:33:46.505729 13516 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 14:33:46.524662 13532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:33:46.526557 13516 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 14:33:46.526741 13516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ts-1/wal
uuid: "ff1fc2db7def4d15a7cc82e9f6ab7eb4"
format_stamp: "Formatted at 2026-06-19 14:33:46 on dist-test-slave-70rm"
server_key: "147066d3877272c23114958578060a3b"
server_key_iv: "a9eec70c2b59d934aacb64a08989d2c1"
server_key_version: "kuduclusterkey@0"
I20260619 14:33:46.526916 13516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:33:46.573596 13516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:33:46.577646 13516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:33:46.577868 13516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:33:46.578614 13516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:33:46.579747 13516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:33:46.579821 13516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:33:46.579882 13516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:33:46.579916 13516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:33:46.591311 13516 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.2:45049
I20260619 14:33:46.591471 13645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.2:45049 every 8 connection(s)
I20260619 14:33:46.592489 13516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
I20260619 14:33:46.596540 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 13516
I20260619 14:33:46.596663 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ts-1/wal/instance
I20260619 14:33:46.596976 2056 external_mini_cluster.cc:1428] Setting key 3e5a4cf9ad5858e81b3ebfaf522c2011
WARNING: no policy specified for kudu/127.2.2.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.3@KRBTEST.COM" created.
Jun 19 14:33:46 dist-test-slave-70rm krb5kdc[12820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879626, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.2@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:33:46.642179 13511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:33:46.594791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:55753 (local address 127.2.2.62:42665)
0619 14:33:46.594972 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 14:33:46.594979 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:33:46.596152 (+ 1173us) server_negotiation.cc:407] Connection header received
0619 14:33:46.597968 (+ 1816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:33:46.597971 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:33:46.598018 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:33:46.598108 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:33:46.599734 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:33:46.600553 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:33:46.624064 (+ 23511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:33:46.624451 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:33:46.624546 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:33:46.633031 (+ 8485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:33:46.633036 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:33:46.633039 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:33:46.633074 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:33:46.634967 (+ 1893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:33:46.640023 (+ 5056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:33:46.640028 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:33:46.640030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:33:46.640147 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:33:46.640681 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:33:46.640685 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:33:46.640687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:33:46.640936 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:33:46.641026 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 14:33:46.641815 (+ 789us) server_negotiation.cc:299] Negotiation successful
0619 14:33:46.641976 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 14:33:46.644241 13648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:33:46.595333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:42665 (local address 127.2.2.2:55753)
0619 14:33:46.595940 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0619 14:33:46.596013 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0619 14:33:46.597696 (+ 1683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:33:46.598382 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:33:46.598393 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:33:46.598812 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:33:46.599540 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:33:46.599564 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:33:46.600729 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:33:46.600734 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:33:46.623811 (+ 23077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:33:46.623837 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:33:46.624751 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:33:46.624828 (+ 77us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:33:46.625060 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:33:46.630529 (+ 5469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:33:46.630555 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:33:46.632765 (+ 2210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:33:46.637615 (+ 4850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:33:46.637623 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:33:46.637646 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:33:46.638076 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:33:46.640283 (+ 2207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:33:46.640286 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:33:46.640288 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:33:46.640522 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:33:46.641101 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:33:46.641110 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:33:46.641494 (+ 384us) client_negotiation.cc:769] Sending connection context
0619 14:33:46.642702 (+ 1208us) client_negotiation.cc:240] Negotiation successful
0619 14:33:46.643112 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":139,"threads_started":1}
I20260619 14:33:46.650741 13646 heartbeater.cc:347] Connected to a master server at 127.2.2.62:42665
I20260619 14:33:46.651068 13646 heartbeater.cc:464] Registering TS with master...
I20260619 14:33:46.651715 13646 heartbeater.cc:511] Master 127.2.2.62:42665 requested a full tablet report, sending...
I20260619 14:33:46.653039 13296 ts_manager.cc:194] Registered new tserver with Master: ff1fc2db7def4d15a7cc82e9f6ab7eb4 (127.2.2.2:45049)
I20260619 14:33:46.653851 13296 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.2.2@KRBTEST.COM'} at 127.2.2.2:55753
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab.
Entry for principal kudu/127.2.2.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab.
I20260619 14:33:46.669241 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.2.20:43581/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.3:0
--local_ip_for_outbound_sockets=127.2.2.3
--webserver_interface=127.2.2.3
--webserver_port=0
--tserver_master_addrs=127.2.2.62:42665
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab
--principal=kudu/127.2.2.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:37049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:33:46 dist-test-slave-70rm krb5kdc[12820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879626, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.62@KRBTEST.COM for HTTP/127.2.2.20@KRBTEST.COM
W20260619 14:33:46.833101 13651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:33:46.833595 13651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:33:46.833730 13651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:33:46.840205 13651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:33:46.840384 13651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:33:46.840535 13651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.3
I20260619 14:33:46.849400 13651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:37049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.2.2.20:43581/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab
--principal=kudu/127.2.2.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.2.3
--webserver_port=0
--tserver_master_addrs=127.2.2.62:42665
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.3
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:33:46.851190 13651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:33:46.852705 13651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:33:46.865424 13659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:33:46.865695 13656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:33:46.865411 13657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:33:46.867256 13651 server_base.cc:1034] running on GCE node
I20260619 14:33:46.867738 13651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:33:46.868582 13651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:33:46.869804 13651 hybrid_clock.cc:648] HybridClock initialized: now 1781879626869783 us; error 35 us; skew 500 ppm
Jun 19 14:33:46 dist-test-slave-70rm krb5kdc[12820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879626, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:33:46.874109 13651 init.cc:377] Logged in from keytab as kudu/127.2.2.3@KRBTEST.COM (short username kudu)
I20260619 14:33:46.875286 13651 webserver.cc:460] Webserver started at http://127.2.2.3:39861/ using document root <none> and password file <none>
I20260619 14:33:46.876000 13651 fs_manager.cc:359] Metadata directory not provided
I20260619 14:33:46.876101 13651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:33:46.876329 13651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 14:33:46 dist-test-slave-70rm krb5kdc[12820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879626, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.3@KRBTEST.COM for HTTP/127.2.2.20@KRBTEST.COM
I20260619 14:33:46.892916 13651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ts-2/data/instance:
uuid: "6bd112c03c824377a22ed3f901f090fe"
format_stamp: "Formatted at 2026-06-19 14:33:46 on dist-test-slave-70rm"
server_key: "75f32a257528e15140d28c88b110cffb"
server_key_iv: "53635928dd4b781ef14627451dd4a8eb"
server_key_version: "kuduclusterkey@0"
I20260619 14:33:46.893826 13651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ts-2/wal/instance:
uuid: "6bd112c03c824377a22ed3f901f090fe"
format_stamp: "Formatted at 2026-06-19 14:33:46 on dist-test-slave-70rm"
server_key: "75f32a257528e15140d28c88b110cffb"
server_key_iv: "53635928dd4b781ef14627451dd4a8eb"
server_key_version: "kuduclusterkey@0"
I20260619 14:33:46.902882 13651 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.000s
I20260619 14:33:46.922585 13666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:33:46.924310 13651 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260619 14:33:46.924456 13651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ts-2/wal
uuid: "6bd112c03c824377a22ed3f901f090fe"
format_stamp: "Formatted at 2026-06-19 14:33:46 on dist-test-slave-70rm"
server_key: "75f32a257528e15140d28c88b110cffb"
server_key_iv: "53635928dd4b781ef14627451dd4a8eb"
server_key_version: "kuduclusterkey@0"
I20260619 14:33:46.924607 13651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:33:46.941975 13651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:33:46.945258 13651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:33:46.945489 13651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:33:46.946173 13651 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:33:46.947086 13651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:33:46.947154 13651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:33:46.947319 13651 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:33:46.947355 13651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:33:46.958585 13651 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.3:39255
I20260619 14:33:46.958806 13779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.3:39255 every 8 connection(s)
I20260619 14:33:46.959583 13651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
I20260619 14:33:46.968158 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 13651
I20260619 14:33:46.968290 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ts-2/wal/instance
I20260619 14:33:46.968545 2056 external_mini_cluster.cc:1428] Setting key 5fd9000f5f02cb7b6af8a6a29b3ae5d1
Jun 19 14:33:46 dist-test-slave-70rm krb5kdc[12820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879626, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.3@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:33:46.975479 13511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:33:46.961881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.3:50849 (local address 127.2.2.62:42665)
0619 14:33:46.962010 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 14:33:46.962015 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:33:46.963019 (+ 1004us) server_negotiation.cc:407] Connection header received
0619 14:33:46.964130 (+ 1111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:33:46.964134 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:33:46.964193 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:33:46.964300 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:33:46.965608 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:33:46.966290 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:33:46.967568 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:33:46.967810 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:33:46.967924 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:33:46.971894 (+ 3970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:33:46.971898 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:33:46.971899 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:33:46.971924 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:33:46.973334 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:33:46.973855 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:33:46.973860 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:33:46.973862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:33:46.973943 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:33:46.974361 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:33:46.974366 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:33:46.974368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:33:46.974554 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:33:46.974667 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 14:33:46.975203 (+ 536us) server_negotiation.cc:299] Negotiation successful
0619 14:33:46.975321 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 14:33:46.976423 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:33:46.962404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:42665 (local address 127.2.2.3:50849)
0619 14:33:46.962864 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0619 14:33:46.962915 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 14:33:46.963911 (+ 996us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:33:46.964503 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:33:46.964512 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:33:46.964905 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:33:46.965456 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:33:46.965469 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:33:46.966408 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:33:46.966413 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:33:46.967440 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:33:46.967449 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:33:46.968952 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:33:46.969020 (+ 68us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:33:46.969230 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:33:46.970161 (+ 931us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:33:46.970182 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:33:46.971718 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:33:46.973464 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:33:46.973470 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:33:46.973488 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:33:46.973752 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:33:46.974038 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:33:46.974042 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:33:46.974044 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:33:46.974269 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:33:46.974667 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:33:46.974675 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:33:46.974957 (+ 282us) client_negotiation.cc:769] Sending connection context
0619 14:33:46.975152 (+ 195us) client_negotiation.cc:240] Negotiation successful
0619 14:33:46.975475 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":131,"threads_started":1}
I20260619 14:33:46.977541 13780 heartbeater.cc:347] Connected to a master server at 127.2.2.62:42665
I20260619 14:33:46.977778 13780 heartbeater.cc:464] Registering TS with master...
I20260619 14:33:46.978343 13780 heartbeater.cc:511] Master 127.2.2.62:42665 requested a full tablet report, sending...
I20260619 14:33:46.979350 13296 ts_manager.cc:194] Registered new tserver with Master: 6bd112c03c824377a22ed3f901f090fe (127.2.2.3:39255)
I20260619 14:33:46.979914 13296 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.2.3@KRBTEST.COM'} at 127.2.2.3:50849
I20260619 14:33:46.983047 2056 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:33:46.992862 13511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:33:46.985060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53824 (local address 127.2.2.62:42665)
0619 14:33:46.985190 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 14:33:46.985194 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:33:46.985424 (+ 230us) server_negotiation.cc:407] Connection header received
0619 14:33:46.985557 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:33:46.985559 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:33:46.985602 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:33:46.985692 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:33:46.986435 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:33:46.986950 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:33:46.987780 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:33:46.988011 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:33:46.988123 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:33:46.989501 (+ 1378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:33:46.989504 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:33:46.989507 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:33:46.989531 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:33:46.991097 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:33:46.991514 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:33:46.991519 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:33:46.991521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:33:46.991614 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:33:46.991884 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:33:46.991889 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:33:46.991890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:33:46.992077 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:33:46.992213 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 14:33:46.992448 (+ 235us) server_negotiation.cc:299] Negotiation successful
0619 14:33:46.992571 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 14:33:46.995911 13296 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53824:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:33:46.999156 13296 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:33:47.023380 13792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:33:47.009941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.3:39255 (local address 127.0.0.1:52408)
0619 14:33:47.017953 (+ 8012us) negotiation.cc:107] Waiting for socket to connect
0619 14:33:47.017970 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:33:47.018086 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:33:47.018910 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:33:47.018913 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:33:47.018930 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:33:47.019036 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:33:47.019042 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:33:47.020418 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:33:47.020421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:33:47.021510 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:33:47.021522 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:33:47.022862 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:33:47.022866 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:33:47.022969 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:33:47.022991 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 14:33:47.023066 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 14:33:47.023211 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7934,"thread_start_us":77,"threads_started":1}
I20260619 14:33:47.023855 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:33:47.008330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55914 (local address 127.2.2.1:36723)
0619 14:33:47.009799 (+ 1469us) server_negotiation.cc:206] Beginning negotiation
0619 14:33:47.009809 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:33:47.010692 (+ 883us) server_negotiation.cc:407] Connection header received
0619 14:33:47.010925 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:33:47.010933 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:33:47.011166 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:33:47.011310 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:33:47.012522 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:33:47.013591 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:33:47.018586 (+ 4995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:33:47.019491 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:33:47.019577 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:33:47.019738 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 14:33:47.023593 (+ 3855us) server_negotiation.cc:299] Negotiation successful
0619 14:33:47.023719 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1352,"thread_start_us":102,"threads_started":1}
I20260619 14:33:47.024353 13797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:33:47.010089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52408 (local address 127.2.2.3:39255)
0619 14:33:47.010486 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0619 14:33:47.010497 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:33:47.018200 (+ 7703us) server_negotiation.cc:407] Connection header received
0619 14:33:47.018350 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:33:47.018356 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:33:47.018609 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:33:47.018764 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:33:47.019158 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:33:47.020289 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:33:47.021670 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:33:47.022528 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:33:47.022649 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:33:47.022797 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 14:33:47.024097 (+ 1300us) server_negotiation.cc:299] Negotiation successful
0619 14:33:47.024209 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":133,"threads_started":1}
I20260619 14:33:47.024788 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:33:47.008822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.2:45049 (local address 127.0.0.1:60274)
0619 14:33:47.011009 (+ 2187us) negotiation.cc:107] Waiting for socket to connect
0619 14:33:47.011021 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:33:47.011099 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:33:47.011921 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:33:47.011925 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:33:47.011941 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:33:47.012049 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:33:47.012056 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:33:47.013514 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:33:47.013517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:33:47.018285 (+ 4768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:33:47.018299 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:33:47.024374 (+ 6075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:33:47.024378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:33:47.024484 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:33:47.024507 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 14:33:47.024585 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 14:33:47.024661 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2131,"mutex_wait_us":57,"thread_start_us":96,"threads_started":1}
I20260619 14:33:47.025211 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:33:47.009225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60274 (local address 127.2.2.2:45049)
0619 14:33:47.011331 (+ 2106us) server_negotiation.cc:206] Beginning negotiation
0619 14:33:47.011343 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:33:47.011364 (+ 21us) server_negotiation.cc:407] Connection header received
0619 14:33:47.011437 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:33:47.011442 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:33:47.011632 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:33:47.011777 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:33:47.012189 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:33:47.013389 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:33:47.022306 (+ 8917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:33:47.023213 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:33:47.023330 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:33:47.023475 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 14:33:47.024942 (+ 1467us) server_negotiation.cc:299] Negotiation successful
0619 14:33:47.025071 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2008,"thread_start_us":106,"threads_started":1}
I20260619 14:33:47.028240 13573 tablet_service.cc:1467] Processing CreateTablet for tablet 275cb738be0c4df99d6850e7f2cb1fa8 (DEFAULT_TABLE table=test-table [id=44deddc2220d4f25814b9d5d5d3cb414]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:33:47.028226 13714 tablet_service.cc:1467] Processing CreateTablet for tablet 275cb738be0c4df99d6850e7f2cb1fa8 (DEFAULT_TABLE table=test-table [id=44deddc2220d4f25814b9d5d5d3cb414]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:33:47.029382 13714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 275cb738be0c4df99d6850e7f2cb1fa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:33:47.029395 13573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 275cb738be0c4df99d6850e7f2cb1fa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:33:47.024296 13796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:33:47.008190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.1:36723 (local address 127.0.0.1:55914)
0619 14:33:47.010537 (+ 2347us) negotiation.cc:107] Waiting for socket to connect
0619 14:33:47.010595 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 14:33:47.010752 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:33:47.011483 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:33:47.011488 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:33:47.011509 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:33:47.012389 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:33:47.012397 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:33:47.013716 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:33:47.013719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:33:47.018452 (+ 4733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:33:47.018459 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:33:47.019847 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:33:47.019851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:33:47.019975 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:33:47.020017 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 14:33:47.024074 (+ 4057us) client_negotiation.cc:240] Negotiation successful
0619 14:33:47.024161 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2172,"mutex_wait_us":21,"thread_start_us":147,"threads_started":1}
I20260619 14:33:47.039450 13799 tablet_bootstrap.cc:492] T 275cb738be0c4df99d6850e7f2cb1fa8 P 6bd112c03c824377a22ed3f901f090fe: Bootstrap starting.
I20260619 14:33:47.041361 13442 tablet_service.cc:1467] Processing CreateTablet for tablet 275cb738be0c4df99d6850e7f2cb1fa8 (DEFAULT_TABLE table=test-table [id=44deddc2220d4f25814b9d5d5d3cb414]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:33:47.042301 13798 tablet_bootstrap.cc:492] T 275cb738be0c4df99d6850e7f2cb1fa8 P ff1fc2db7def4d15a7cc82e9f6ab7eb4: Bootstrap starting.
I20260619 14:33:47.042541 13442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 275cb738be0c4df99d6850e7f2cb1fa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:33:47.045156 13798 tablet_bootstrap.cc:654] T 275cb738be0c4df99d6850e7f2cb1fa8 P ff1fc2db7def4d15a7cc82e9f6ab7eb4: Neither blocks nor log segments found. Creating new log.
I20260619 14:33:47.045748 13799 tablet_bootstrap.cc:654] T 275cb738be0c4df99d6850e7f2cb1fa8 P 6bd112c03c824377a22ed3f901f090fe: Neither blocks nor log segments found. Creating new log.
I20260619 14:33:47.046732 13799 log.cc:824] T 275cb738be0c4df99d6850e7f2cb1fa8 P 6bd112c03c824377a22ed3f901f090fe: Log is configured to *not* fsync() on all Append() calls
I20260619 14:33:47.048082 13798 log.cc:824] T 275cb738be0c4df99d6850e7f2cb1fa8 P ff1fc2db7def4d15a7cc82e9f6ab7eb4: Log is configured to *not* fsync() on all Append() calls
I20260619 14:33:47.050352 13798 tablet_bootstrap.cc:492] T 275cb738be0c4df99d6850e7f2cb1fa8 P ff1fc2db7def4d15a7cc82e9f6ab7eb4: No bootstrap required, opened a new log
I20260619 14:33:47.050560 13798 ts_tablet_manager.cc:1397] T 275cb738be0c4df99d6850e7f2cb1fa8 P ff1fc2db7def4d15a7cc82e9f6ab7eb4: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 14:33:47.056069 13799 tablet_bootstrap.cc:492] T 275cb738be0c4df99d6850e7f2cb1fa8 P 6bd112c03c824377a22ed3f901f090fe: No bootstrap required, opened a new log
I20260619 14:33:47.056289 13799 ts_tablet_manager.cc:1397] T 275cb738be0c4df99d6850e7f2cb1fa8 P 6bd112c03c824377a22ed3f901f090fe: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.000s
I20260619 14:33:47.058437 13798 raft_consensus.cc:348] T 275cb738be0c4df99d6850e7f2cb1fa8 P ff1fc2db7def4d15a7cc82e9f6ab7eb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef8bc152baa4506bf365e3f76304b75" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 36723 } } peers { permanent_uuid: "ff1fc2db7def4d15a7cc82e9f6ab7eb4" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 45049 } } peers { permanent_uuid: "6bd112c03c824377a22ed3f901f090fe" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 39255 } }
I20260619 14:33:47.058813 13798 raft_consensus.cc:374] T 275cb738be0c4df99d6850e7f2cb1fa8 P ff1fc2db7def4d15a7cc82e9f6ab7eb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:33:47.058918 13798 raft_consensus.cc:729] T 275cb738be0c4df99d6850e7f2cb1fa8 P ff1fc2db7def4d15a7cc82e9f6ab7eb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff1fc2db7def4d15a7cc82e9f6ab7eb4, State: Initialized, Role: FOLLOWER
I20260619 14:33:47.059542 13798 consensus_queue.cc:260] T 275cb738be0c4df99d6850e7f2cb1fa8 P ff1fc2db7def4d15a7cc82e9f6ab7eb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef8bc152baa4506bf365e3f76304b75" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 36723 } } peers { permanent_uuid: "ff1fc2db7def4d15a7cc82e9f6ab7eb4" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 45049 } } peers { permanent_uuid: "6bd112c03c824377a22ed3f901f090fe" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 39255 } }
I20260619 14:33:47.061129 13799 raft_consensus.cc:348] T 275cb738be0c4df99d6850e7f2cb1fa8 P 6bd112c03c824377a22ed3f901f090fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef8bc152baa4506bf365e3f76304b75" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 36723 } } peers { permanent_uuid: "ff1fc2db7def4d15a7cc82e9f6ab7eb4" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 45049 } } peers { permanent_uuid: "6bd112c03c824377a22ed3f901f090fe" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 39255 } }
I20260619 14:33:47.061340 13798 ts_tablet_manager.cc:1428] T 275cb738be0c4df99d6850e7f2cb1fa8 P ff1fc2db7def4d15a7cc82e9f6ab7eb4: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20260619 14:33:47.061447 13799 raft_consensus.cc:374] T 275cb738be0c4df99d6850e7f2cb1fa8 P 6bd112c03c824377a22ed3f901f090fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:33:47.061561 13799 raft_consensus.cc:729] T 275cb738be0c4df99d6850e7f2cb1fa8 P 6bd112c03c824377a22ed3f901f090fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bd112c03c824377a22ed3f901f090fe, State: Initialized, Role: FOLLOWER
I20260619 14:33:47.061834 13646 heartbeater.cc:503] Master 127.2.2.62:42665 was elected leader, sending a full tablet report...
I20260619 14:33:47.062098 13799 consensus_queue.cc:260] T 275cb738be0c4df99d6850e7f2cb1fa8 P 6bd112c03c824377a22ed3f901f090fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef8bc152baa4506bf365e3f76304b75" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 36723 } } peers { permanent_uuid: "ff1fc2db7def4d15a7cc82e9f6ab7eb4" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 45049 } } peers { permanent_uuid: "6bd112c03c824377a22ed3f901f090fe" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 39255 } }
I20260619 14:33:47.063429 13799 ts_tablet_manager.cc:1428] T 275cb738be0c4df99d6850e7f2cb1fa8 P 6bd112c03c824377a22ed3f901f090fe: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20260619 14:33:47.063802 13780 heartbeater.cc:503] Master 127.2.2.62:42665 was elected leader, sending a full tablet report...
I20260619 14:33:47.063781 13802 tablet_bootstrap.cc:492] T 275cb738be0c4df99d6850e7f2cb1fa8 P eef8bc152baa4506bf365e3f76304b75: Bootstrap starting.
I20260619 14:33:47.066763 13802 tablet_bootstrap.cc:654] T 275cb738be0c4df99d6850e7f2cb1fa8 P eef8bc152baa4506bf365e3f76304b75: Neither blocks nor log segments found. Creating new log.
I20260619 14:33:47.068174 13802 log.cc:824] T 275cb738be0c4df99d6850e7f2cb1fa8 P eef8bc152baa4506bf365e3f76304b75: Log is configured to *not* fsync() on all Append() calls
I20260619 14:33:47.071209 13802 tablet_bootstrap.cc:492] T 275cb738be0c4df99d6850e7f2cb1fa8 P eef8bc152baa4506bf365e3f76304b75: No bootstrap required, opened a new log
I20260619 14:33:47.071457 13802 ts_tablet_manager.cc:1397] T 275cb738be0c4df99d6850e7f2cb1fa8 P eef8bc152baa4506bf365e3f76304b75: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 14:33:47.076157 13802 raft_consensus.cc:348] T 275cb738be0c4df99d6850e7f2cb1fa8 P eef8bc152baa4506bf365e3f76304b75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef8bc152baa4506bf365e3f76304b75" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 36723 } } peers { permanent_uuid: "ff1fc2db7def4d15a7cc82e9f6ab7eb4" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 45049 } } peers { permanent_uuid: "6bd112c03c824377a22ed3f901f090fe" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 39255 } }
I20260619 14:33:47.076490 13802 raft_consensus.cc:374] T 275cb738be0c4df99d6850e7f2cb1fa8 P eef8bc152baa4506bf365e3f76304b75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:33:47.076644 13802 raft_consensus.cc:729] T 275cb738be0c4df99d6850e7f2cb1fa8 P eef8bc152baa4506bf365e3f76304b75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eef8bc152baa4506bf365e3f76304b75, State: Initialized, Role: FOLLOWER
I20260619 14:33:47.077224 13802 consensus_queue.cc:260] T 275cb738be0c4df99d6850e7f2cb1fa8 P eef8bc152baa4506bf365e3f76304b75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef8bc152baa4506bf365e3f76304b75" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 36723 } } peers { permanent_uuid: "ff1fc2db7def4d15a7cc82e9f6ab7eb4" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 45049 } } peers { permanent_uuid: "6bd112c03c824377a22ed3f901f090fe" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 39255 } }
I20260619 14:33:47.078388 13802 ts_tablet_manager.cc:1428] T 275cb738be0c4df99d6850e7f2cb1fa8 P eef8bc152baa4506bf365e3f76304b75: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260619 14:33:47.078825 13508 heartbeater.cc:503] Master 127.2.2.62:42665 was elected leader, sending a full tablet report...
W20260619 14:33:47.100626 13647 tablet.cc:2369] T 275cb738be0c4df99d6850e7f2cb1fa8 P ff1fc2db7def4d15a7cc82e9f6ab7eb4: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 14:33:47 dist-test-slave-70rm krb5kdc[12820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879626, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.62@KRBTEST.COM for HTTP/127.2.2.20@KRBTEST.COM
I20260619 14:33:47.143975 13803 raft_consensus.cc:482] T 275cb738be0c4df99d6850e7f2cb1fa8 P ff1fc2db7def4d15a7cc82e9f6ab7eb4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:33:47.144222 13803 raft_consensus.cc:504] T 275cb738be0c4df99d6850e7f2cb1fa8 P ff1fc2db7def4d15a7cc82e9f6ab7eb4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef8bc152baa4506bf365e3f76304b75" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 36723 } } peers { permanent_uuid: "ff1fc2db7def4d15a7cc82e9f6ab7eb4" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 45049 } } peers { permanent_uuid: "6bd112c03c824377a22ed3f901f090fe" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 39255 } }
I20260619 14:33:47.145625 13803 leader_election.cc:288] T 275cb738be0c4df99d6850e7f2cb1fa8 P ff1fc2db7def4d15a7cc82e9f6ab7eb4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eef8bc152baa4506bf365e3f76304b75 (127.2.2.1:36723), 6bd112c03c824377a22ed3f901f090fe (127.2.2.3:39255)
I20260619 14:33:47.151208 13648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:33:47.145843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.1:36723 (local address 127.2.2.2:50339)
0619 14:33:47.146826 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0619 14:33:47.146840 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:33:47.146911 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:33:47.147272 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:33:47.147275 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:33:47.147293 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:33:47.147415 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:33:47.147423 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:33:47.148442 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:33:47.148446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:33:47.149655 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:33:47.149668 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:33:47.150751 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:33:47.150754 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:33:47.150874 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:33:47.150897 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 14:33:47.150973 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 14:33:47.151062 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":904}
I20260619 14:33:47.152290 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:33:47.146391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:50339 (local address 127.2.2.1:36723)
0619 14:33:47.147066 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0619 14:33:47.147069 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:33:47.147081 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:33:47.147123 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:33:47.147125 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:33:47.147165 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:33:47.147250 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:33:47.147551 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:33:47.148305 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:33:47.149826 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:33:47.150503 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:33:47.150614 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:33:47.150685 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 14:33:47.152032 (+ 1347us) server_negotiation.cc:299] Negotiation successful
0619 14:33:47.152166 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603}
I20260619 14:33:47.152894 13455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "275cb738be0c4df99d6850e7f2cb1fa8" candidate_uuid: "ff1fc2db7def4d15a7cc82e9f6ab7eb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eef8bc152baa4506bf365e3f76304b75" is_pre_election: true
I20260619 14:33:47.153249 13455 raft_consensus.cc:2454] T 275cb738be0c4df99d6850e7f2cb1fa8 P eef8bc152baa4506bf365e3f76304b75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff1fc2db7def4d15a7cc82e9f6ab7eb4 in term 0.
I20260619 14:33:47.154006 13536 leader_election.cc:302] T 275cb738be0c4df99d6850e7f2cb1fa8 P ff1fc2db7def4d15a7cc82e9f6ab7eb4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eef8bc152baa4506bf365e3f76304b75, ff1fc2db7def4d15a7cc82e9f6ab7eb4; no voters:
I20260619 14:33:47.154322 13797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:33:47.146394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:39069 (local address 127.2.2.3:39255)
0619 14:33:47.146526 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 14:33:47.146531 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:33:47.150012 (+ 3481us) server_negotiation.cc:407] Connection header received
0619 14:33:47.150273 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:33:47.150277 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:33:47.150331 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:33:47.150380 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:33:47.150820 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:33:47.151535 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:33:47.153000 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:33:47.153585 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:33:47.153660 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:33:47.153748 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 14:33:47.154107 (+ 359us) server_negotiation.cc:299] Negotiation successful
0619 14:33:47.154202 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 14:33:47.154395 13803 raft_consensus.cc:2781] T 275cb738be0c4df99d6850e7f2cb1fa8 P ff1fc2db7def4d15a7cc82e9f6ab7eb4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:33:47.154484 13803 raft_consensus.cc:482] T 275cb738be0c4df99d6850e7f2cb1fa8 P ff1fc2db7def4d15a7cc82e9f6ab7eb4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:33:47.154531 13803 raft_consensus.cc:3037] T 275cb738be0c4df99d6850e7f2cb1fa8 P ff1fc2db7def4d15a7cc82e9f6ab7eb4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:33:47.154764 13809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:33:47.145967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.3:39255 (local address 127.2.2.2:39069)
0619 14:33:47.149893 (+ 3926us) negotiation.cc:107] Waiting for socket to connect
0619 14:33:47.149908 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:33:47.150151 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:33:47.150525 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:33:47.150528 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:33:47.150546 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:33:47.150655 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:33:47.150661 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:33:47.151704 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:33:47.151707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:33:47.152856 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:33:47.152867 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:33:47.153876 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:33:47.153879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:33:47.153977 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:33:47.153998 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 14:33:47.154538 (+ 540us) client_negotiation.cc:240] Negotiation successful
0619 14:33:47.154605 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3850,"thread_start_us":115,"threads_started":1}
I20260619 14:33:47.155880 13803 raft_consensus.cc:504] T 275cb738be0c4df99d6850e7f2cb1fa8 P ff1fc2db7def4d15a7cc82e9f6ab7eb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef8bc152baa4506bf365e3f76304b75" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 36723 } } peers { permanent_uuid: "ff1fc2db7def4d15a7cc82e9f6ab7eb4" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 45049 } } peers { permanent_uuid: "6bd112c03c824377a22ed3f901f090fe" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 39255 } }
I20260619 14:33:47.156315 13803 leader_election.cc:288] T 275cb738be0c4df99d6850e7f2cb1fa8 P ff1fc2db7def4d15a7cc82e9f6ab7eb4 [CANDIDATE]: Term 1 election: Requested vote from peers eef8bc152baa4506bf365e3f76304b75 (127.2.2.1:36723), 6bd112c03c824377a22ed3f901f090fe (127.2.2.3:39255)
I20260619 14:33:47.156759 13734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "275cb738be0c4df99d6850e7f2cb1fa8" candidate_uuid: "ff1fc2db7def4d15a7cc82e9f6ab7eb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bd112c03c824377a22ed3f901f090fe" is_pre_election: true
I20260619 14:33:47.157008 13734 raft_consensus.cc:2454] T 275cb738be0c4df99d6850e7f2cb1fa8 P 6bd112c03c824377a22ed3f901f090fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff1fc2db7def4d15a7cc82e9f6ab7eb4 in term 0.
I20260619 14:33:47.157922 13734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "275cb738be0c4df99d6850e7f2cb1fa8" candidate_uuid: "ff1fc2db7def4d15a7cc82e9f6ab7eb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bd112c03c824377a22ed3f901f090fe"
I20260619 14:33:47.158038 13734 raft_consensus.cc:3037] T 275cb738be0c4df99d6850e7f2cb1fa8 P 6bd112c03c824377a22ed3f901f090fe [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:33:47.158852 13455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "275cb738be0c4df99d6850e7f2cb1fa8" candidate_uuid: "ff1fc2db7def4d15a7cc82e9f6ab7eb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eef8bc152baa4506bf365e3f76304b75"
I20260619 14:33:47.158980 13455 raft_consensus.cc:3037] T 275cb738be0c4df99d6850e7f2cb1fa8 P eef8bc152baa4506bf365e3f76304b75 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:33:47.159516 13734 raft_consensus.cc:2454] T 275cb738be0c4df99d6850e7f2cb1fa8 P 6bd112c03c824377a22ed3f901f090fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff1fc2db7def4d15a7cc82e9f6ab7eb4 in term 1.
I20260619 14:33:47.159935 13533 leader_election.cc:302] T 275cb738be0c4df99d6850e7f2cb1fa8 P ff1fc2db7def4d15a7cc82e9f6ab7eb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bd112c03c824377a22ed3f901f090fe, ff1fc2db7def4d15a7cc82e9f6ab7eb4; no voters:
I20260619 14:33:47.160125 13803 raft_consensus.cc:2781] T 275cb738be0c4df99d6850e7f2cb1fa8 P ff1fc2db7def4d15a7cc82e9f6ab7eb4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:33:47.160354 13803 raft_consensus.cc:686] T 275cb738be0c4df99d6850e7f2cb1fa8 P ff1fc2db7def4d15a7cc82e9f6ab7eb4 [term 1 LEADER]: Becoming Leader. State: Replica: ff1fc2db7def4d15a7cc82e9f6ab7eb4, State: Running, Role: LEADER
I20260619 14:33:47.160394 13455 raft_consensus.cc:2454] T 275cb738be0c4df99d6850e7f2cb1fa8 P eef8bc152baa4506bf365e3f76304b75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff1fc2db7def4d15a7cc82e9f6ab7eb4 in term 1.
I20260619 14:33:47.160758 13803 consensus_queue.cc:237] T 275cb738be0c4df99d6850e7f2cb1fa8 P ff1fc2db7def4d15a7cc82e9f6ab7eb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef8bc152baa4506bf365e3f76304b75" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 36723 } } peers { permanent_uuid: "ff1fc2db7def4d15a7cc82e9f6ab7eb4" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 45049 } } peers { permanent_uuid: "6bd112c03c824377a22ed3f901f090fe" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 39255 } }
I20260619 14:33:47.164532 13295 catalog_manager.cc:5495] T 275cb738be0c4df99d6850e7f2cb1fa8 P ff1fc2db7def4d15a7cc82e9f6ab7eb4 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff1fc2db7def4d15a7cc82e9f6ab7eb4 (127.2.2.2). New cstate: current_term: 1 leader_uuid: "ff1fc2db7def4d15a7cc82e9f6ab7eb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef8bc152baa4506bf365e3f76304b75" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 36723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff1fc2db7def4d15a7cc82e9f6ab7eb4" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 45049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6bd112c03c824377a22ed3f901f090fe" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 39255 } health_report { overall_health: UNKNOWN } } }
W20260619 14:33:47.198688 13509 tablet.cc:2369] T 275cb738be0c4df99d6850e7f2cb1fa8 P eef8bc152baa4506bf365e3f76304b75: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:33:47.211040 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:33:47.205524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60288 (local address 127.2.2.2:45049)
0619 14:33:47.206018 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0619 14:33:47.206023 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:33:47.206036 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:33:47.206142 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:33:47.206145 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:33:47.206218 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:33:47.206322 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:33:47.206752 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:33:47.207482 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:33:47.208777 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:33:47.209108 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:33:47.209213 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:33:47.209643 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:33:47.210214 (+ 571us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:33:47.210330 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 14:33:47.210809 (+ 479us) server_negotiation.cc:299] Negotiation successful
0619 14:33:47.210872 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390}
W20260619 14:33:47.214159 13781 tablet.cc:2369] T 275cb738be0c4df99d6850e7f2cb1fa8 P 6bd112c03c824377a22ed3f901f090fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:33:47.231986 13734 raft_consensus.cc:1261] T 275cb738be0c4df99d6850e7f2cb1fa8 P 6bd112c03c824377a22ed3f901f090fe [term 1 FOLLOWER]: Refusing update from remote peer ff1fc2db7def4d15a7cc82e9f6ab7eb4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:33:47.232956 13803 consensus_queue.cc:1035] T 275cb738be0c4df99d6850e7f2cb1fa8 P ff1fc2db7def4d15a7cc82e9f6ab7eb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bd112c03c824377a22ed3f901f090fe" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 39255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:33:47.239449 13455 raft_consensus.cc:1261] T 275cb738be0c4df99d6850e7f2cb1fa8 P eef8bc152baa4506bf365e3f76304b75 [term 1 FOLLOWER]: Refusing update from remote peer ff1fc2db7def4d15a7cc82e9f6ab7eb4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:33:47.240386 13803 consensus_queue.cc:1035] T 275cb738be0c4df99d6850e7f2cb1fa8 P ff1fc2db7def4d15a7cc82e9f6ab7eb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eef8bc152baa4506bf365e3f76304b75" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 36723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:33:47.252286 13813 mvcc.cc:205] Tried to move back new op lower bound from 7298578953096196096 to 7298578952853741568. Current Snapshot: MvccSnapshot[applied={T|T < 7298578953096196096}]
I20260619 14:33:47.254917 13816 mvcc.cc:205] Tried to move back new op lower bound from 7298578953096196096 to 7298578952853741568. Current Snapshot: MvccSnapshot[applied={T|T < 7298578953096196096}]
I20260619 14:33:47.279080 13296 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:33:47.279271 13296 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:33:47.281965 13296 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1c7f8dcef5ec43b6b5057c86646d62eb: Sending DeleteTablet for 3 replicas of tablet 275cb738be0c4df99d6850e7f2cb1fa8
I20260619 14:33:47.283705 13442 tablet_service.cc:1514] Processing DeleteTablet for tablet 275cb738be0c4df99d6850e7f2cb1fa8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:33:47 UTC) from {username='kudu', principal='kudu/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:55914
I20260619 14:33:47.284437 13714 tablet_service.cc:1514] Processing DeleteTablet for tablet 275cb738be0c4df99d6850e7f2cb1fa8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:33:47 UTC) from {username='kudu', principal='kudu/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:52408
I20260619 14:33:47.285982 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 13377
I20260619 14:33:47.291276 13573 tablet_service.cc:1514] Processing DeleteTablet for tablet 275cb738be0c4df99d6850e7f2cb1fa8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:33:47 UTC) from {username='kudu', principal='kudu/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:60274
I20260619 14:33:47.297497 13823 tablet_replica.cc:330] T 275cb738be0c4df99d6850e7f2cb1fa8 P 6bd112c03c824377a22ed3f901f090fe: stopping tablet replica
I20260619 14:33:47.302515 13823 raft_consensus.cc:2229] T 275cb738be0c4df99d6850e7f2cb1fa8 P 6bd112c03c824377a22ed3f901f090fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:33:47.303016 13823 raft_consensus.cc:2258] T 275cb738be0c4df99d6850e7f2cb1fa8 P 6bd112c03c824377a22ed3f901f090fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:33:47.303323 13824 tablet_replica.cc:330] T 275cb738be0c4df99d6850e7f2cb1fa8 P ff1fc2db7def4d15a7cc82e9f6ab7eb4: stopping tablet replica
I20260619 14:33:47.303684 13824 raft_consensus.cc:2229] T 275cb738be0c4df99d6850e7f2cb1fa8 P ff1fc2db7def4d15a7cc82e9f6ab7eb4 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:33:47.304479 13824 raft_consensus.cc:2258] T 275cb738be0c4df99d6850e7f2cb1fa8 P ff1fc2db7def4d15a7cc82e9f6ab7eb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:33:47.305549 13823 ts_tablet_manager.cc:1905] T 275cb738be0c4df99d6850e7f2cb1fa8 P 6bd112c03c824377a22ed3f901f090fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:33:47.306186 13824 ts_tablet_manager.cc:1905] T 275cb738be0c4df99d6850e7f2cb1fa8 P ff1fc2db7def4d15a7cc82e9f6ab7eb4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 14:33:47.306901 13283 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.2.1:36723 (error 108)
I20260619 14:33:47.308586 13823 ts_tablet_manager.cc:1918] T 275cb738be0c4df99d6850e7f2cb1fa8 P 6bd112c03c824377a22ed3f901f090fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:33:47.308699 13823 log.cc:1196] T 275cb738be0c4df99d6850e7f2cb1fa8 P 6bd112c03c824377a22ed3f901f090fe: Deleting WAL directory at /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879514993456-2056-0/minicluster-data/ts-2/wal/wals/275cb738be0c4df99d6850e7f2cb1fa8
I20260619 14:33:47.308882 13824 ts_tablet_manager.cc:1918] T 275cb738be0c4df99d6850e7f2cb1fa8 P ff1fc2db7def4d15a7cc82e9f6ab7eb4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:33:47.308926 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 13516
I20260619 14:33:47.312321 13796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:33:47.311997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.1:36723 (local address 127.0.0.1:55926)
0619 14:33:47.312172 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 14:33:47.312236 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.1:36723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
W20260619 14:33:47.312577 13283 catalog_manager.cc:4584] TS eef8bc152baa4506bf365e3f76304b75 (127.2.2.1:36723): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 275cb738be0c4df99d6850e7f2cb1fa8: Network error: Client connection negotiation failed: client connection to 127.2.2.1:36723: connect: Connection refused (error 111)
I20260619 14:33:47.312971 13823 ts_tablet_manager.cc:1939] T 275cb738be0c4df99d6850e7f2cb1fa8 P 6bd112c03c824377a22ed3f901f090fe: Deleting consensus metadata
I20260619 14:33:47.314589 13280 catalog_manager.cc:4841] TS 6bd112c03c824377a22ed3f901f090fe (127.2.2.3:39255): tablet 275cb738be0c4df99d6850e7f2cb1fa8 (table test-table [id=44deddc2220d4f25814b9d5d5d3cb414]) successfully deleted
W20260619 14:33:47.320184 13796 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:33:47.319427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.2:45049 (local address 127.0.0.1:60298)
0619 14:33:47.319624 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0619 14:33:47.319640 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:33:47.319733 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:33:47.320068 (+ 335us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.2:45049: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":109}
I20260619 14:33:47.320782 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 13651
I20260619 14:33:47.328675 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 13264
I20260619 14:33:47.336844 2056 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 14:33:47.671696 2056 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 14:33:47.671828 2056 mini_ranger.cc:66] Stopping Ranger...
I20260619 14:33:48.016625 2056 mini_ranger.cc:68] Stopped Ranger
2026-06-19 14:33:48.016 UTC [12850] LOG: received smart shutdown request
2026-06-19 14:33:48.020 UTC [12850] LOG: background worker "logical replication launcher" (PID 12859) exited with exit code 1
2026-06-19 14:33:48.020 UTC [12854] LOG: shutting down
2026-06-19 14:33:48.188 UTC [12850] LOG: database system is shut down
2026-06-19T14:33:48Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46657 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:33:48 dist-test-slave-70rm krb5kdc[13834](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:33:48 dist-test-slave-70rm krb5kdc[13834](info): set up 2 sockets
Jun 19 14:33:48 dist-test-slave-70rm krb5kdc[13834](info): commencing operation
krb5kdc: starting...
W20260619 14:33:49.346306 2056 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Jun 19 14:33:49 dist-test-slave-70rm krb5kdc[13834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:33:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:33:49Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.2.2.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.2.2.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.2.2.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/krb5kdc/rangeradmin_127.2.2.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.2.2.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/krb5kdc/rangeradmin_127.2.2.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.2.2.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.2.2.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.2.2.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/krb5kdc/rangerlookup_127.2.2.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.2.2.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/krb5kdc/rangerlookup_127.2.2.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.2.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.2.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.2.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/krb5kdc/HTTP_127.2.2.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.2.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/krb5kdc/HTTP_127.2.2.20@KRBTEST.COM.keytab.
I20260619 14:33:49.513160 2056 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskMyx_Zw/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/postgres -l logfile start
2026-06-19 14:33:52.057 UTC [13864] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 14:33:52.057 UTC [13864] LOG: listening on IPv4 address "127.2.2.20", port 54037
2026-06-19 14:33:52.061 UTC [13864] LOG: listening on Unix socket "/tmp/.s.PGSQL.54037"
2026-06-19 14:33:52.082 UTC [13867] LOG: database system was shut down at 2026-06-19 14:33:50 UTC
2026-06-19 14:33:52.086 UTC [13864] LOG: database system is ready to accept connections
I20260619 14:33:53.054020 2056 mini_postgres.cc:96] Postgres bound to 54037
2026-06-19 14:33:53.060 UTC [13875] FATAL: database "slave" does not exist
127.2.2.20:54037 - accepting connections
I20260619 14:33:53.060957 2056 mini_ranger.cc:161] Starting Ranger...
I20260619 14:33:53.075975 2056 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 14:33:53.557605 2056 mini_ranger.cc:87] Created ranger Postgres database
I20260619 14:33:53.557729 2056 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ranger-admin
2026-06-19 14:33:53,827 [I] DB FLAVOR :POSTGRES
2026-06-19 14:33:53,827 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 14:33:53,827 [I] Checking connection..
2026-06-19 14:33:53,828 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMyx_Zw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMyx_Zw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.2.20:54037/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 14:33:54,098 [I] Checking connection passed.
2026-06-19 14:33:54,098 [I] --------- Verifying version history table ---------
2026-06-19 14:33:54,098 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMyx_Zw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMyx_Zw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.2.20:54037/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 14:33:54,372 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 14:33:54,373 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMyx_Zw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMyx_Zw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.2.20:54037/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 14:33:54,633 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 14:33:54,633 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 14:33:54,633 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMyx_Zw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMyx_Zw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.2.20:54037/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskMyx_Zw/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 14:33:54.890 UTC [13939] WARNING: there is no transaction in progress
2026-06-19 14:33:54,897 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 14:33:54,897 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMyx_Zw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMyx_Zw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.2.20:54037/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 14:33:55,143 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 14:33:55,143 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 14:33:55,143 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMyx_Zw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMyx_Zw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.2.20:54037/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 14:33:55,402 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMyx_Zw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMyx_Zw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.2.20:54037/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 14:33:55,649 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMyx_Zw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMyx_Zw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.2.20:54037/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-70rm.c.gcp-upstream.internal','N') ;"
2026-06-19 14:33:55,883 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMyx_Zw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMyx_Zw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.2.20:54037/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 14:33:56,146 [I] Table x_portal_user does not exist in database ranger
2026-06-19 14:33:56,146 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMyx_Zw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMyx_Zw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.2.20:54037/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 14:33:56,377 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 14:33:56,378 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMyx_Zw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMyx_Zw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.2.20:54037/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 14:33:56,624 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMyx_Zw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMyx_Zw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.2.20:54037/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 14:33:56,852 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 14:33:56,852 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMyx_Zw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMyx_Zw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.2.20:54037/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskMyx_Zw/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 14:33:57.827 UTC [14047] WARNING: there is no transaction in progress
2026-06-19 14:33:57.843 UTC [14047] WARNING: there is no transaction in progress
2026-06-19 14:33:57.859 UTC [14047] WARNING: there is no transaction in progress
2026-06-19 14:33:57.875 UTC [14047] WARNING: there is no transaction in progress
2026-06-19 14:33:57.892 UTC [14047] WARNING: there is no transaction in progress
2026-06-19 14:33:57.910 UTC [14047] WARNING: there is no transaction in progress
2026-06-19 14:33:57.968 UTC [14047] WARNING: there is no transaction in progress
2026-06-19 14:33:57.975 UTC [14047] WARNING: there is no transaction in progress
2026-06-19 14:33:57.989 UTC [14047] WARNING: there is no transaction in progress
2026-06-19 14:33:58.000 UTC [14047] WARNING: there is no transaction in progress
2026-06-19 14:33:58.011 UTC [14047] WARNING: there is no transaction in progress
2026-06-19 14:33:58.025 UTC [14047] WARNING: there is no transaction in progress
2026-06-19 14:33:58.035 UTC [14047] WARNING: there is no transaction in progress
2026-06-19 14:33:58.042 UTC [14047] WARNING: there is no transaction in progress
2026-06-19 14:33:58.050 UTC [14047] WARNING: there is no transaction in progress
2026-06-19 14:33:58.057 UTC [14047] WARNING: there is no transaction in progress
2026-06-19 14:33:58,699 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 14:33:58,700 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMyx_Zw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMyx_Zw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.2.20:54037/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 14:33:58,941 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 14:33:58,942 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMyx_Zw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMyx_Zw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.2.20:54037/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 14:33:59,193 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 14:33:59,193 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMyx_Zw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMyx_Zw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.2.20:54037/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 14:33:59,430 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 14:33:59,431 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMyx_Zw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMyx_Zw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.2.20:54037/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 14:33:59,696 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMyx_Zw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMyx_Zw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.2.20:54037/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 14:33:59,931 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMyx_Zw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMyx_Zw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.2.20:54037/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-70rm.c.gcp-upstream.internal';"
2026-06-19 14:34:00,160 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 14:34:00,161 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMyx_Zw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMyx_Zw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.2.20:54037/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 14:34:00,397 [I] DB_PATCHES have already been applied
I20260619 14:34:00.403227 2056 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ranger-admin:/tmp/dist-test-taskMyx_Zw/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskMyx_Zw/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskMyx_Zw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMyx_Zw/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 14:34:00.403326 2056 mini_ranger.cc:193] Using host: 127.2.2.20
I20260619 14:34:00.406714 2056 mini_ranger.cc:227] Ranger admin URL: http://127.2.2.20:44169
Jun 19, 2026 2:34:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 2:34:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 2:34:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 2:34:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 2:34:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 2:34:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 2:34:01 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 14:34:01.522037 2056 mini_ranger.cc:160] Time spent starting Ranger: real 8.461s user 0.000s sys 0.010s
Jun 19, 2026 2:34:21 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 2:34:21 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 19, 2026 2:34:21 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 2:34:21 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 2:34:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 2:34:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 2:34:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 2:34:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 2:34:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 2:34:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 2:34:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 2:34:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 2:34:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 2:34:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 2:34:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 2:34:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 2:34:21 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 2:34:22 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 14:34:23.399324 2056 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.2.2.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.2.2.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.2.2.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/krb5kdc/rangerkms_127.2.2.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.2.2.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/krb5kdc/rangerkms_127.2.2.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.2.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.2.2.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.2.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/krb5kdc/HTTP_127.2.2.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.2.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/krb5kdc/HTTP_127.2.2.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 14:34:23 dist-test-slave-70rm krb5kdc[13834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879663, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 14:34:23.489740 2056 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 14:34:23.499899 2056 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 14:34:24.533318 2056 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 14:34:24.533411 2056 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ranger-kms
I20260619 14:34:24.533434 2056 mini_ranger_kms.cc:227] Using postgres at 127.2.2.20:54037
2026-06-19 14:34:24,705 [I] DB FLAVOR :POSTGRES
2026-06-19 14:34:24,706 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 14:34:24,706 [I] Checking connection
2026-06-19 14:34:24,706 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMyx_Zw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.2.20:54037/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 14:34:24,954 [I] connection success
2026-06-19 14:34:24,954 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 14:34:24,954 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 14:34:24,954 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMyx_Zw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.2.20:54037/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 14:34:25,197 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 14:34:25,197 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 14:34:25,197 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 14:34:25,198 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMyx_Zw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.2.20:54037/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 14:34:25,488 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 14:34:26.003582 2056 mini_ranger_kms.cc:310] Created kms service
I20260619 14:34:26.215467 2056 mini_ranger_kms.cc:326] Created kudu user
I20260619 14:34:26.267407 2056 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 14:34:26 dist-test-slave-70rm krb5kdc[13834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879663, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.2.2.20@KRBTEST.COM
I20260619 14:34:26.703668 2056 mini_ranger_kms.cc:384] Added ranger policy
I20260619 14:34:26.703905 2056 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ranger-kms:/tmp/dist-test-taskMyx_Zw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMyx_Zw/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskMyx_Zw/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 14:34:26.703933 2056 mini_ranger_kms.cc:240] Using host: 127.2.2.20
I20260619 14:34:26.707543 2056 mini_ranger_kms.cc:276] Ranger KMS PID: 14227
I20260619 14:34:26.707629 2056 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.2.2.20:59627
Jun 19, 2026 2:34:27 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 2:34:27 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 2:34:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 2:34:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 2:34:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 2:34:28 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 14:34:28.944664 2056 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.455s user 0.017s sys 0.139s
I20260619 14:34:28.944811 2056 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 14:34:28.944898 2056 mini_ranger_kms.cc:401] 127.2.2.20:59627/kms/v1/keys
Jun 19 14:34:29 dist-test-slave-70rm krb5kdc[13834](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781879669, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.2.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 14:34:29 dist-test-slave-70rm krb5kdc[13834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879669, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.2.20@KRBTEST.COM for HTTP/127.2.2.20@KRBTEST.COM
Jun 19 14:34:29 dist-test-slave-70rm krb5kdc[13834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879669, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.2.20@KRBTEST.COM for HTTP/127.2.2.20@KRBTEST.COM
Jun 19 14:34:32 dist-test-slave-70rm krb5kdc[13834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.2.2.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab.
Entry for principal kudu/127.2.2.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab.
I20260619 14:34:32.278097 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:42457
--webserver_interface=127.2.2.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:36237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.2.62:42457
--ranger_config_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.2.20:59627/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:34:32.399663 14278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:34:32.400018 14278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:34:32.400102 14278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:34:32.404379 14278 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:34:32.404490 14278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:34:32.404524 14278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:34:32.404579 14278 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:34:32.404616 14278 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:34:32.410497 14278 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:36237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/master-0/wal
--ranger_kms_url=127.2.2.20:59627/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.2.62:42457
--ranger_config_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:42457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.2.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14278
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:34:32.411645 14278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:34:32.412693 14278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:34:32.422434 14286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:34:32.422451 14283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:34:32.422444 14284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:34:32.422596 14278 server_base.cc:1034] running on GCE node
I20260619 14:34:32.424696 14278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:34:32.425860 14278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:34:32.427074 14278 hybrid_clock.cc:648] HybridClock initialized: now 1781879672427051 us; error 50 us; skew 500 ppm
Jun 19 14:34:32 dist-test-slave-70rm krb5kdc[13834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879672, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:34:32.430024 14278 init.cc:377] Logged in from keytab as kudu/127.2.2.62@KRBTEST.COM (short username kudu)
I20260619 14:34:32.431046 14278 webserver.cc:460] Webserver started at http://127.2.2.62:39635/ using document root <none> and password file <none>
I20260619 14:34:32.431610 14278 fs_manager.cc:359] Metadata directory not provided
I20260619 14:34:32.431690 14278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:34:32.431910 14278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 14:34:32 dist-test-slave-70rm krb5kdc[13834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879672, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.62@KRBTEST.COM for HTTP/127.2.2.20@KRBTEST.COM
I20260619 14:34:32.487104 14278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/master-0/data/instance:
uuid: "839a0162d61f4ee88b66ed6c3717f9e0"
format_stamp: "Formatted at 2026-06-19 14:34:32 on dist-test-slave-70rm"
server_key: "0d5c19e2b299df77adb85f2ea51a2162"
server_key_iv: "e91fa8bd51124a301fc944767f68366c"
server_key_version: "kuduclusterkey@0"
I20260619 14:34:32.487852 14278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/master-0/wal/instance:
uuid: "839a0162d61f4ee88b66ed6c3717f9e0"
format_stamp: "Formatted at 2026-06-19 14:34:32 on dist-test-slave-70rm"
server_key: "0d5c19e2b299df77adb85f2ea51a2162"
server_key_iv: "e91fa8bd51124a301fc944767f68366c"
server_key_version: "kuduclusterkey@0"
I20260619 14:34:32.492858 14278 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 14:34:32.510814 14293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:34:32.512377 14278 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:34:32.512529 14278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/master-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/master-0/wal
uuid: "839a0162d61f4ee88b66ed6c3717f9e0"
format_stamp: "Formatted at 2026-06-19 14:34:32 on dist-test-slave-70rm"
server_key: "0d5c19e2b299df77adb85f2ea51a2162"
server_key_iv: "e91fa8bd51124a301fc944767f68366c"
server_key_version: "kuduclusterkey@0"
I20260619 14:34:32.512673 14278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:34:32.529393 14278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:34:32.532455 14278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:34:32.532668 14278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:34:32.541168 14278 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.62:42457
I20260619 14:34:32.541188 14345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.62:42457 every 8 connection(s)
I20260619 14:34:32.542328 14278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
I20260619 14:34:32.545264 14346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:34:32.546872 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 14278
I20260619 14:34:32.547003 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/master-0/wal/instance
I20260619 14:34:32.547305 2056 external_mini_cluster.cc:1428] Setting key 277633c898b3f55d879275048f300b48
I20260619 14:34:32.552501 14346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 839a0162d61f4ee88b66ed6c3717f9e0: Bootstrap starting.
I20260619 14:34:32.555207 14346 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 839a0162d61f4ee88b66ed6c3717f9e0: Neither blocks nor log segments found. Creating new log.
I20260619 14:34:32.556106 14346 log.cc:824] T 00000000000000000000000000000000 P 839a0162d61f4ee88b66ed6c3717f9e0: Log is configured to *not* fsync() on all Append() calls
Jun 19 14:34:32 dist-test-slave-70rm krb5kdc[13834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:34:32.558537 14346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 839a0162d61f4ee88b66ed6c3717f9e0: No bootstrap required, opened a new log
I20260619 14:34:32.563087 14346 raft_consensus.cc:348] T 00000000000000000000000000000000 P 839a0162d61f4ee88b66ed6c3717f9e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "839a0162d61f4ee88b66ed6c3717f9e0" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 42457 } }
I20260619 14:34:32.563352 14346 raft_consensus.cc:374] T 00000000000000000000000000000000 P 839a0162d61f4ee88b66ed6c3717f9e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:34:32.563426 14346 raft_consensus.cc:729] T 00000000000000000000000000000000 P 839a0162d61f4ee88b66ed6c3717f9e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 839a0162d61f4ee88b66ed6c3717f9e0, State: Initialized, Role: FOLLOWER
I20260619 14:34:32.564013 14346 consensus_queue.cc:260] T 00000000000000000000000000000000 P 839a0162d61f4ee88b66ed6c3717f9e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "839a0162d61f4ee88b66ed6c3717f9e0" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 42457 } }
I20260619 14:34:32.564304 14346 raft_consensus.cc:388] T 00000000000000000000000000000000 P 839a0162d61f4ee88b66ed6c3717f9e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:34:32.564385 14346 raft_consensus.cc:482] T 00000000000000000000000000000000 P 839a0162d61f4ee88b66ed6c3717f9e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:34:32.564508 14346 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 839a0162d61f4ee88b66ed6c3717f9e0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:34:32.565450 14349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:32.548869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38326 (local address 127.2.2.62:42457)
0619 14:34:32.549355 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0619 14:34:32.549375 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 14:34:32.549421 (+ 46us) server_negotiation.cc:407] Connection header received
0619 14:34:32.550338 (+ 917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:34:32.550367 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:34:32.550837 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:34:32.551288 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:34:32.552138 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:32.553596 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:32.554918 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:32.555650 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:32.555805 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:32.558943 (+ 3138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:34:32.558949 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:34:32.558967 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:34:32.558998 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:34:32.561673 (+ 2675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:32.562269 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:32.562275 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:32.562282 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:32.562422 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:32.562756 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:32.562759 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:32.562761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:32.563317 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:34:32.563525 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0619 14:34:32.564016 (+ 491us) server_negotiation.cc:299] Negotiation successful
0619 14:34:32.564476 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":138,"threads_started":1}
I20260619 14:34:32.565661 14346 raft_consensus.cc:504] T 00000000000000000000000000000000 P 839a0162d61f4ee88b66ed6c3717f9e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "839a0162d61f4ee88b66ed6c3717f9e0" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 42457 } }
I20260619 14:34:32.566022 14346 leader_election.cc:302] T 00000000000000000000000000000000 P 839a0162d61f4ee88b66ed6c3717f9e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 839a0162d61f4ee88b66ed6c3717f9e0; no voters:
I20260619 14:34:32.566431 14346 leader_election.cc:288] T 00000000000000000000000000000000 P 839a0162d61f4ee88b66ed6c3717f9e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:34:32.566546 14351 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 839a0162d61f4ee88b66ed6c3717f9e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:34:32.566763 14351 raft_consensus.cc:686] T 00000000000000000000000000000000 P 839a0162d61f4ee88b66ed6c3717f9e0 [term 1 LEADER]: Becoming Leader. State: Replica: 839a0162d61f4ee88b66ed6c3717f9e0, State: Running, Role: LEADER
I20260619 14:34:32.567097 14351 consensus_queue.cc:237] T 00000000000000000000000000000000 P 839a0162d61f4ee88b66ed6c3717f9e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "839a0162d61f4ee88b66ed6c3717f9e0" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 42457 } }
I20260619 14:34:32.567772 14346 sys_catalog.cc:564] T 00000000000000000000000000000000 P 839a0162d61f4ee88b66ed6c3717f9e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:34:32.568377 14351 sys_catalog.cc:455] T 00000000000000000000000000000000 P 839a0162d61f4ee88b66ed6c3717f9e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "839a0162d61f4ee88b66ed6c3717f9e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "839a0162d61f4ee88b66ed6c3717f9e0" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 42457 } } }
I20260619 14:34:32.568514 14351 sys_catalog.cc:458] T 00000000000000000000000000000000 P 839a0162d61f4ee88b66ed6c3717f9e0 [sys.catalog]: This master's current role is: LEADER
I20260619 14:34:32.568753 14351 sys_catalog.cc:455] T 00000000000000000000000000000000 P 839a0162d61f4ee88b66ed6c3717f9e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 839a0162d61f4ee88b66ed6c3717f9e0. Latest consensus state: current_term: 1 leader_uuid: "839a0162d61f4ee88b66ed6c3717f9e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "839a0162d61f4ee88b66ed6c3717f9e0" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 42457 } } }
I20260619 14:34:32.568915 14351 sys_catalog.cc:458] T 00000000000000000000000000000000 P 839a0162d61f4ee88b66ed6c3717f9e0 [sys.catalog]: This master's current role is: LEADER
I20260619 14:34:32.569199 14358 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:34:32.572562 14358 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:34:32.573082 14346 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 14:34:32.584038 14358 catalog_manager.cc:1347] Generated new cluster ID: 0769d42f4a45406db14a3f73e0a2c1e3
I20260619 14:34:32.584210 14358 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:34:32.614195 14358 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:34:32.615666 14358 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:34:32.624366 14358 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 839a0162d61f4ee88b66ed6c3717f9e0: Generated new TSK 0
I20260619 14:34:32.625398 14358 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.2.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab.
Entry for principal kudu/127.2.2.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab.
I20260619 14:34:33.375299 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.2.20:59627/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.1:0
--local_ip_for_outbound_sockets=127.2.2.1
--webserver_interface=127.2.2.1
--webserver_port=0
--tserver_master_addrs=127.2.2.62:42457
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab
--principal=kudu/127.2.2.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:36237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:34:33.521219 14391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:34:33.521605 14391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:34:33.521682 14391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:34:33.527869 14391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:34:33.527977 14391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:34:33.528069 14391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.1
I20260619 14:34:33.534278 14391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:36237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.2.2.20:59627/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab
--principal=kudu/127.2.2.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.2.1
--webserver_port=0
--tserver_master_addrs=127.2.2.62:42457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.1
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:34:33.535493 14391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:34:33.536739 14391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:34:33.548504 14399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:34:33.548595 14398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:34:33.548717 14401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:34:33.550021 14391 server_base.cc:1034] running on GCE node
I20260619 14:34:33.550730 14391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:34:33.551581 14391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:34:33.552760 14391 hybrid_clock.cc:648] HybridClock initialized: now 1781879673552734 us; error 51 us; skew 500 ppm
Jun 19 14:34:33 dist-test-slave-70rm krb5kdc[13834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879673, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:34:33.556214 14391 init.cc:377] Logged in from keytab as kudu/127.2.2.1@KRBTEST.COM (short username kudu)
I20260619 14:34:33.557260 14391 webserver.cc:460] Webserver started at http://127.2.2.1:35945/ using document root <none> and password file <none>
I20260619 14:34:33.557847 14391 fs_manager.cc:359] Metadata directory not provided
I20260619 14:34:33.557912 14391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:34:33.558260 14391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 14:34:33 dist-test-slave-70rm krb5kdc[13834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879673, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.1@KRBTEST.COM for HTTP/127.2.2.20@KRBTEST.COM
I20260619 14:34:33.575134 14391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-0/data/instance:
uuid: "d4f0800abcef4b468e118c96c42a3c2d"
format_stamp: "Formatted at 2026-06-19 14:34:33 on dist-test-slave-70rm"
server_key: "0a1fa5a1c6e609b1b4a832efc04babd5"
server_key_iv: "6364b2586031276b9c2b51fd449d955e"
server_key_version: "kuduclusterkey@0"
I20260619 14:34:33.576253 14391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-0/wal/instance:
uuid: "d4f0800abcef4b468e118c96c42a3c2d"
format_stamp: "Formatted at 2026-06-19 14:34:33 on dist-test-slave-70rm"
server_key: "0a1fa5a1c6e609b1b4a832efc04babd5"
server_key_iv: "6364b2586031276b9c2b51fd449d955e"
server_key_version: "kuduclusterkey@0"
I20260619 14:34:33.581571 14391 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 14:34:33.602553 14408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:34:33.604414 14391 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.004s sys 0.000s
I20260619 14:34:33.604633 14391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-0/wal
uuid: "d4f0800abcef4b468e118c96c42a3c2d"
format_stamp: "Formatted at 2026-06-19 14:34:33 on dist-test-slave-70rm"
server_key: "0a1fa5a1c6e609b1b4a832efc04babd5"
server_key_iv: "6364b2586031276b9c2b51fd449d955e"
server_key_version: "kuduclusterkey@0"
I20260619 14:34:33.604867 14391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:34:33.637463 14391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:34:33.641516 14391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:34:33.641804 14391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:34:33.642601 14391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:34:33.643875 14391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:34:33.643989 14391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:34:33.644122 14391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:34:33.644202 14391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:34:33.655496 14391 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.1:32785
I20260619 14:34:33.655871 14523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.1:32785 every 8 connection(s)
I20260619 14:34:33.656698 14391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
I20260619 14:34:33.665160 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 14391
I20260619 14:34:33.665303 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-0/wal/instance
I20260619 14:34:33.665645 2056 external_mini_cluster.cc:1428] Setting key 20358f8beccc239b9e8218c5ea6181ff
Jun 19 14:34:33 dist-test-slave-70rm krb5kdc[13834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879673, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.1@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:34:33.672124 14526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:33.658751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:33405 (local address 127.2.2.62:42457)
0619 14:34:33.659050 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0619 14:34:33.659055 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:34:33.659855 (+ 800us) server_negotiation.cc:407] Connection header received
0619 14:34:33.660876 (+ 1021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:34:33.660879 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:34:33.660937 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:34:33.661061 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:34:33.662244 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:33.662871 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:33.664207 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:33.664439 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:33.664547 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:33.667835 (+ 3288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:34:33.667839 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:34:33.667842 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:34:33.667871 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:34:33.669815 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:33.670463 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:33.670467 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:33.670469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:33.670554 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:33.670933 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:33.670936 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:33.670938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:33.671140 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:34:33.671244 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 14:34:33.671805 (+ 561us) server_negotiation.cc:299] Negotiation successful
0619 14:34:33.671940 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":97,"threads_started":1}
I20260619 14:34:33.673106 14527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:33.659211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:42457 (local address 127.2.2.1:33405)
0619 14:34:33.659691 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:33.659748 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 14:34:33.660660 (+ 912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:34:33.661260 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:34:33.661267 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:34:33.661547 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:34:33.662045 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:33.662058 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:33.663005 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:33.663008 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:33.664061 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:33.664071 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:33.664545 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:33.664606 (+ 61us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:33.664836 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:33.665721 (+ 885us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:34:33.665742 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:34:33.667632 (+ 1890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:34:33.669942 (+ 2310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:34:33.669949 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:34:33.669966 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:34:33.670349 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:34:33.670661 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:34:33.670664 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:34:33.670666 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:34:33.670831 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:34:33.671255 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:34:33.671264 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:34:33.671549 (+ 285us) client_negotiation.cc:769] Sending connection context
0619 14:34:33.671753 (+ 204us) client_negotiation.cc:240] Negotiation successful
0619 14:34:33.672092 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":144,"threads_started":1}
I20260619 14:34:33.674584 14524 heartbeater.cc:347] Connected to a master server at 127.2.2.62:42457
I20260619 14:34:33.674846 14524 heartbeater.cc:464] Registering TS with master...
I20260619 14:34:33.675496 14524 heartbeater.cc:511] Master 127.2.2.62:42457 requested a full tablet report, sending...
I20260619 14:34:33.677074 14310 ts_manager.cc:194] Registered new tserver with Master: d4f0800abcef4b468e118c96c42a3c2d (127.2.2.1:32785)
I20260619 14:34:33.678328 14310 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.2.1@KRBTEST.COM'} at 127.2.2.1:33405
WARNING: no policy specified for kudu/127.2.2.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab.
Entry for principal kudu/127.2.2.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab.
I20260619 14:34:33.709998 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.2.20:59627/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.2:0
--local_ip_for_outbound_sockets=127.2.2.2
--webserver_interface=127.2.2.2
--webserver_port=0
--tserver_master_addrs=127.2.2.62:42457
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab
--principal=kudu/127.2.2.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:36237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:34:33 dist-test-slave-70rm krb5kdc[13834](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781879673, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 14:34:33.874183 14530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:34:33.874557 14530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:34:33.874624 14530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:34:33.880859 14530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:34:33.881032 14530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:34:33.881176 14530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.2
I20260619 14:34:33.889583 14530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:36237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.2.2.20:59627/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab
--principal=kudu/127.2.2.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.2.2
--webserver_port=0
--tserver_master_addrs=127.2.2.62:42457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.2
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:34:33.891310 14530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:34:33.892753 14530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:34:33.905759 14537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:34:33.905759 14539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:34:33.906391 14536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:34:33.907387 14530 server_base.cc:1034] running on GCE node
I20260619 14:34:33.907900 14530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:34:33.908774 14530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:34:33.910028 14530 hybrid_clock.cc:648] HybridClock initialized: now 1781879673909998 us; error 58 us; skew 500 ppm
Jun 19 14:34:33 dist-test-slave-70rm krb5kdc[13834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879673, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:34:33.913756 14530 init.cc:377] Logged in from keytab as kudu/127.2.2.2@KRBTEST.COM (short username kudu)
I20260619 14:34:33.915128 14530 webserver.cc:460] Webserver started at http://127.2.2.2:43233/ using document root <none> and password file <none>
I20260619 14:34:33.915891 14530 fs_manager.cc:359] Metadata directory not provided
I20260619 14:34:33.915997 14530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:34:33.916275 14530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 14:34:33 dist-test-slave-70rm krb5kdc[13834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879673, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.2@KRBTEST.COM for HTTP/127.2.2.20@KRBTEST.COM
I20260619 14:34:33.932320 14530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-1/data/instance:
uuid: "bdddfb4b708d4441b93ae829351f92e1"
format_stamp: "Formatted at 2026-06-19 14:34:33 on dist-test-slave-70rm"
server_key: "1c8264203e9ec933410a810eddcd8613"
server_key_iv: "c401c763d1f3378b18803189880aeafa"
server_key_version: "kuduclusterkey@0"
I20260619 14:34:33.935323 14530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-1/wal/instance:
uuid: "bdddfb4b708d4441b93ae829351f92e1"
format_stamp: "Formatted at 2026-06-19 14:34:33 on dist-test-slave-70rm"
server_key: "1c8264203e9ec933410a810eddcd8613"
server_key_iv: "c401c763d1f3378b18803189880aeafa"
server_key_version: "kuduclusterkey@0"
I20260619 14:34:33.941119 14530 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 14:34:33.966509 14546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:34:33.968380 14530 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 14:34:33.968528 14530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-1/wal
uuid: "bdddfb4b708d4441b93ae829351f92e1"
format_stamp: "Formatted at 2026-06-19 14:34:33 on dist-test-slave-70rm"
server_key: "1c8264203e9ec933410a810eddcd8613"
server_key_iv: "c401c763d1f3378b18803189880aeafa"
server_key_version: "kuduclusterkey@0"
I20260619 14:34:33.968689 14530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:34:33.984215 14530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:34:33.987439 14530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:34:33.987655 14530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:34:33.988373 14530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:34:33.989339 14530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:34:33.989410 14530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:34:33.989506 14530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:34:33.989552 14530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:34:34.000869 14530 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.2:33091
I20260619 14:34:34.000891 14659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.2:33091 every 8 connection(s)
I20260619 14:34:34.001997 14530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
I20260619 14:34:34.004433 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 14530
I20260619 14:34:34.004570 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-1/wal/instance
I20260619 14:34:34.004904 2056 external_mini_cluster.cc:1428] Setting key 36a84e0a14b4e3196b20ab24f7e7ac39
Jun 19 14:34:34 dist-test-slave-70rm krb5kdc[13834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879673, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.2@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:34:34.021030 14526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:34.005395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:51135 (local address 127.2.2.62:42457)
0619 14:34:34.005825 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0619 14:34:34.005830 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:34:34.006382 (+ 552us) server_negotiation.cc:407] Connection header received
0619 14:34:34.007643 (+ 1261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:34:34.007650 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:34:34.007711 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:34:34.007768 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:34:34.009600 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:34.010459 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:34.012408 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:34.012759 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:34.012844 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:34.016218 (+ 3374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:34:34.016223 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:34:34.016226 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:34:34.016259 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:34:34.018267 (+ 2008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:34.019005 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:34.019009 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:34.019012 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:34.019107 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:34.019571 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:34.019574 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:34.019576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:34.019827 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:34:34.019903 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 14:34:34.020686 (+ 783us) server_negotiation.cc:299] Negotiation successful
0619 14:34:34.020830 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332}
I20260619 14:34:34.023093 14662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:34.004832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:42457 (local address 127.2.2.2:51135)
0619 14:34:34.006219 (+ 1387us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:34.006283 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 14:34:34.007386 (+ 1103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:34:34.008231 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:34:34.008244 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:34:34.008632 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:34:34.009391 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:34.009413 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:34.010696 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:34.010699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:34.012253 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:34.012265 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:34.013022 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:34.013087 (+ 65us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:34.013295 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:34.014063 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:34:34.014087 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:34:34.015905 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:34:34.018441 (+ 2536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:34:34.018448 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:34:34.018467 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:34:34.018875 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:34:34.019227 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:34:34.019230 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:34:34.019232 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:34:34.019451 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:34:34.019990 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:34:34.019996 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:34:34.020371 (+ 375us) client_negotiation.cc:769] Sending connection context
0619 14:34:34.021531 (+ 1160us) client_negotiation.cc:240] Negotiation successful
0619 14:34:34.021924 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1122,"thread_start_us":180,"threads_started":1}
I20260619 14:34:34.024684 14660 heartbeater.cc:347] Connected to a master server at 127.2.2.62:42457
I20260619 14:34:34.024957 14660 heartbeater.cc:464] Registering TS with master...
I20260619 14:34:34.025590 14660 heartbeater.cc:511] Master 127.2.2.62:42457 requested a full tablet report, sending...
I20260619 14:34:34.026952 14310 ts_manager.cc:194] Registered new tserver with Master: bdddfb4b708d4441b93ae829351f92e1 (127.2.2.2:33091)
I20260619 14:34:34.027733 14310 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.2.2@KRBTEST.COM'} at 127.2.2.2:51135
WARNING: no policy specified for kudu/127.2.2.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab.
Entry for principal kudu/127.2.2.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab.
I20260619 14:34:34.061560 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.2.20:59627/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.3:0
--local_ip_for_outbound_sockets=127.2.2.3
--webserver_interface=127.2.2.3
--webserver_port=0
--tserver_master_addrs=127.2.2.62:42457
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab
--principal=kudu/127.2.2.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:36237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:34:34 dist-test-slave-70rm krb5kdc[13834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879673, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.62@KRBTEST.COM for HTTP/127.2.2.20@KRBTEST.COM
W20260619 14:34:34.293745 14665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:34:34.294374 14665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:34:34.294525 14665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:34:34.302273 14665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:34:34.302501 14665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:34:34.302656 14665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.3
I20260619 14:34:34.312188 14665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:36237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.2.2.20:59627/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab
--principal=kudu/127.2.2.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.2.3
--webserver_port=0
--tserver_master_addrs=127.2.2.62:42457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.3
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:34:34.314067 14665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:34:34.315981 14665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:34:34.330843 14673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:34:34.330991 14670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:34:34.330863 14671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:34:34.332832 14665 server_base.cc:1034] running on GCE node
I20260619 14:34:34.333411 14665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:34:34.334337 14665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:34:34.335711 14665 hybrid_clock.cc:648] HybridClock initialized: now 1781879674335638 us; error 94 us; skew 500 ppm
Jun 19 14:34:34 dist-test-slave-70rm krb5kdc[13834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879674, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:34:34.339722 14665 init.cc:377] Logged in from keytab as kudu/127.2.2.3@KRBTEST.COM (short username kudu)
I20260619 14:34:34.340981 14665 webserver.cc:460] Webserver started at http://127.2.2.3:41579/ using document root <none> and password file <none>
I20260619 14:34:34.341806 14665 fs_manager.cc:359] Metadata directory not provided
I20260619 14:34:34.341960 14665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:34:34.342275 14665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 14:34:34 dist-test-slave-70rm krb5kdc[13834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879674, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.3@KRBTEST.COM for HTTP/127.2.2.20@KRBTEST.COM
I20260619 14:34:34.362916 14665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-2/data/instance:
uuid: "bd5b0d612d214923b64c57044de4b6d0"
format_stamp: "Formatted at 2026-06-19 14:34:34 on dist-test-slave-70rm"
server_key: "db9598549446f5d279d7968288f4a7dc"
server_key_iv: "333a7f583fc4a8ebac94cbf81cac0dab"
server_key_version: "kuduclusterkey@0"
I20260619 14:34:34.364166 14665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-2/wal/instance:
uuid: "bd5b0d612d214923b64c57044de4b6d0"
format_stamp: "Formatted at 2026-06-19 14:34:34 on dist-test-slave-70rm"
server_key: "db9598549446f5d279d7968288f4a7dc"
server_key_iv: "333a7f583fc4a8ebac94cbf81cac0dab"
server_key_version: "kuduclusterkey@0"
I20260619 14:34:34.371730 14665 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.000s
I20260619 14:34:34.402685 14680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:34:34.404762 14665 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.002s sys 0.000s
I20260619 14:34:34.404917 14665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-2/wal
uuid: "bd5b0d612d214923b64c57044de4b6d0"
format_stamp: "Formatted at 2026-06-19 14:34:34 on dist-test-slave-70rm"
server_key: "db9598549446f5d279d7968288f4a7dc"
server_key_iv: "333a7f583fc4a8ebac94cbf81cac0dab"
server_key_version: "kuduclusterkey@0"
I20260619 14:34:34.405064 14665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:34:34.453899 14665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:34:34.468632 14665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:34:34.468950 14665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:34:34.469980 14665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:34:34.471244 14665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:34:34.471369 14665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:34:34.471483 14665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:34:34.471561 14665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:34:34.495471 14665 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.3:39967
I20260619 14:34:34.496927 14665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
I20260619 14:34:34.505422 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 14665
I20260619 14:34:34.505571 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-2/wal/instance
I20260619 14:34:34.505901 2056 external_mini_cluster.cc:1428] Setting key f1bfb27ebe6cdff853fdbca8a2de8df6
I20260619 14:34:34.511096 14793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.3:39967 every 8 connection(s)
Jun 19 14:34:34 dist-test-slave-70rm krb5kdc[13834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879674, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.3@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:34:34.541035 14526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:34.514771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.3:44115 (local address 127.2.2.62:42457)
0619 14:34:34.514887 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 14:34:34.514892 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:34:34.518643 (+ 3751us) server_negotiation.cc:407] Connection header received
0619 14:34:34.526281 (+ 7638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:34:34.526286 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:34:34.526358 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:34:34.526416 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:34:34.528331 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:34.529136 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:34.530977 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:34.531306 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:34.531377 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:34.536432 (+ 5055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:34:34.536437 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:34:34.536440 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:34:34.536472 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:34:34.538323 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:34.539094 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:34.539098 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:34.539100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:34.539194 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:34.539648 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:34.539651 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:34.539653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:34.539887 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:34:34.539969 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 14:34:34.540726 (+ 757us) server_negotiation.cc:299] Negotiation successful
0619 14:34:34.540874 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 14:34:34.543123 14796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:34.513556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:42457 (local address 127.2.2.3:44115)
0619 14:34:34.518454 (+ 4898us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:34.518524 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0619 14:34:34.519641 (+ 1117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:34:34.526943 (+ 7302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:34:34.526955 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:34:34.527363 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:34:34.528130 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:34.528152 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:34.529338 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:34.529341 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:34.530824 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:34.530837 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:34.531540 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:34.531605 (+ 65us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:34.531831 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:34.534334 (+ 2503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:34:34.534359 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:34:34.536171 (+ 1812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:34:34.538517 (+ 2346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:34:34.538524 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:34:34.538545 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:34:34.538960 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:34:34.539308 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:34:34.539311 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:34:34.539313 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:34:34.539526 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:34:34.540039 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:34:34.540046 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:34:34.540419 (+ 373us) client_negotiation.cc:769] Sending connection context
0619 14:34:34.541550 (+ 1131us) client_negotiation.cc:240] Negotiation successful
0619 14:34:34.541957 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4636,"thread_start_us":168,"threads_started":1}
I20260619 14:34:34.555977 14794 heartbeater.cc:347] Connected to a master server at 127.2.2.62:42457
I20260619 14:34:34.556305 14794 heartbeater.cc:464] Registering TS with master...
I20260619 14:34:34.556967 14794 heartbeater.cc:511] Master 127.2.2.62:42457 requested a full tablet report, sending...
I20260619 14:34:34.558621 14310 ts_manager.cc:194] Registered new tserver with Master: bd5b0d612d214923b64c57044de4b6d0 (127.2.2.3:39967)
I20260619 14:34:34.559456 14310 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.2.3@KRBTEST.COM'} at 127.2.2.3:44115
I20260619 14:34:34.572139 2056 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:34:34.573063 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 14391
I20260619 14:34:34.582062 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 14530
I20260619 14:34:34.590179 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 14665
I20260619 14:34:34.598831 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 14278
I20260619 14:34:34.604622 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:42457
--webserver_interface=127.2.2.62
--webserver_port=39635
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:36237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.2.62:42457
--ranger_config_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.2.20:59627/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.2.20:59627/kms with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:34:34.719643 14797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:34:34.719933 14797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:34:34.719974 14797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:34:34.724193 14797 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:34:34.724264 14797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:34:34.724287 14797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:34:34.724305 14797 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:34:34.724320 14797 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:34:34.729899 14797 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:36237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.2.20:59627/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.2.62:42457
--ranger_config_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:42457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.2.62
--webserver_port=39635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:34:34.731153 14797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:34:34.732223 14797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:34:34.741075 14802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:34:34.741065 14805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:34:34.741065 14803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:34:34.741197 14797 server_base.cc:1034] running on GCE node
I20260619 14:34:34.742013 14797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:34:34.743206 14797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:34:34.744472 14797 hybrid_clock.cc:648] HybridClock initialized: now 1781879674744457 us; error 36 us; skew 500 ppm
Jun 19 14:34:34 dist-test-slave-70rm krb5kdc[13834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879674, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:34:34.747532 14797 init.cc:377] Logged in from keytab as kudu/127.2.2.62@KRBTEST.COM (short username kudu)
I20260619 14:34:34.748454 14797 webserver.cc:460] Webserver started at http://127.2.2.62:39635/ using document root <none> and password file <none>
I20260619 14:34:34.749025 14797 fs_manager.cc:359] Metadata directory not provided
I20260619 14:34:34.749125 14797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 14:34:34 dist-test-slave-70rm krb5kdc[13834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879674, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.62@KRBTEST.COM for HTTP/127.2.2.20@KRBTEST.COM
I20260619 14:34:34.799978 14797 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:34:34.802512 14813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:34:34.804088 14797 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:34:34.804229 14797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/master-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/master-0/wal
uuid: "839a0162d61f4ee88b66ed6c3717f9e0"
format_stamp: "Formatted at 2026-06-19 14:34:32 on dist-test-slave-70rm"
server_key: "0d5c19e2b299df77adb85f2ea51a2162"
server_key_iv: "e91fa8bd51124a301fc944767f68366c"
server_key_version: "kuduclusterkey@0"
I20260619 14:34:34.804941 14797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:34:34.828609 14797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:34:34.831899 14797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:34:34.832069 14797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:34:34.839608 14797 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.62:42457
I20260619 14:34:34.839609 14865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.62:42457 every 8 connection(s)
I20260619 14:34:34.840781 14797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
I20260619 14:34:34.842499 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 14797
I20260619 14:34:34.842947 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.2.20:59627/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.1:32785
--local_ip_for_outbound_sockets=127.2.2.1
--tserver_master_addrs=127.2.2.62:42457
--webserver_port=35945
--webserver_interface=127.2.2.1
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab
--principal=kudu/127.2.2.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:36237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.2.20:59627/kms with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:34:34.845319 14866 sys_catalog.cc:263] Verifying existing consensus state
I20260619 14:34:34.847143 14866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 839a0162d61f4ee88b66ed6c3717f9e0: Bootstrap starting.
I20260619 14:34:34.864069 14866 log.cc:824] T 00000000000000000000000000000000 P 839a0162d61f4ee88b66ed6c3717f9e0: Log is configured to *not* fsync() on all Append() calls
I20260619 14:34:34.871667 14866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 839a0162d61f4ee88b66ed6c3717f9e0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:34:34.872131 14866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 839a0162d61f4ee88b66ed6c3717f9e0: Bootstrap complete.
I20260619 14:34:34.876412 14866 raft_consensus.cc:348] T 00000000000000000000000000000000 P 839a0162d61f4ee88b66ed6c3717f9e0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "839a0162d61f4ee88b66ed6c3717f9e0" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 42457 } }
I20260619 14:34:34.876869 14866 raft_consensus.cc:729] T 00000000000000000000000000000000 P 839a0162d61f4ee88b66ed6c3717f9e0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 839a0162d61f4ee88b66ed6c3717f9e0, State: Initialized, Role: FOLLOWER
I20260619 14:34:34.877493 14866 consensus_queue.cc:260] T 00000000000000000000000000000000 P 839a0162d61f4ee88b66ed6c3717f9e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "839a0162d61f4ee88b66ed6c3717f9e0" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 42457 } }
I20260619 14:34:34.877650 14866 raft_consensus.cc:388] T 00000000000000000000000000000000 P 839a0162d61f4ee88b66ed6c3717f9e0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:34:34.877746 14866 raft_consensus.cc:482] T 00000000000000000000000000000000 P 839a0162d61f4ee88b66ed6c3717f9e0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:34:34.877869 14866 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 839a0162d61f4ee88b66ed6c3717f9e0 [term 1 FOLLOWER]: Advancing to term 2
I20260619 14:34:34.879954 14866 raft_consensus.cc:504] T 00000000000000000000000000000000 P 839a0162d61f4ee88b66ed6c3717f9e0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "839a0162d61f4ee88b66ed6c3717f9e0" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 42457 } }
I20260619 14:34:34.880311 14866 leader_election.cc:302] T 00000000000000000000000000000000 P 839a0162d61f4ee88b66ed6c3717f9e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 839a0162d61f4ee88b66ed6c3717f9e0; no voters:
I20260619 14:34:34.880666 14866 leader_election.cc:288] T 00000000000000000000000000000000 P 839a0162d61f4ee88b66ed6c3717f9e0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 14:34:34.880832 14870 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 839a0162d61f4ee88b66ed6c3717f9e0 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 14:34:34.881129 14870 raft_consensus.cc:686] T 00000000000000000000000000000000 P 839a0162d61f4ee88b66ed6c3717f9e0 [term 2 LEADER]: Becoming Leader. State: Replica: 839a0162d61f4ee88b66ed6c3717f9e0, State: Running, Role: LEADER
I20260619 14:34:34.881551 14870 consensus_queue.cc:237] T 00000000000000000000000000000000 P 839a0162d61f4ee88b66ed6c3717f9e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "839a0162d61f4ee88b66ed6c3717f9e0" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 42457 } }
I20260619 14:34:34.881959 14866 sys_catalog.cc:564] T 00000000000000000000000000000000 P 839a0162d61f4ee88b66ed6c3717f9e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:34:34.883275 14872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 839a0162d61f4ee88b66ed6c3717f9e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 839a0162d61f4ee88b66ed6c3717f9e0. Latest consensus state: current_term: 2 leader_uuid: "839a0162d61f4ee88b66ed6c3717f9e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "839a0162d61f4ee88b66ed6c3717f9e0" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 42457 } } }
I20260619 14:34:34.883308 14871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 839a0162d61f4ee88b66ed6c3717f9e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "839a0162d61f4ee88b66ed6c3717f9e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "839a0162d61f4ee88b66ed6c3717f9e0" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 42457 } } }
I20260619 14:34:34.883404 14872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 839a0162d61f4ee88b66ed6c3717f9e0 [sys.catalog]: This master's current role is: LEADER
I20260619 14:34:34.883414 14871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 839a0162d61f4ee88b66ed6c3717f9e0 [sys.catalog]: This master's current role is: LEADER
I20260619 14:34:34.883764 14879 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:34:34.887384 14879 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:34:34.887941 14866 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 14:34:34.888165 14879 catalog_manager.cc:1259] Loaded cluster ID: 0769d42f4a45406db14a3f73e0a2c1e3
I20260619 14:34:34.888218 14879 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:34:34.893178 14879 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:34:34.895147 14879 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 839a0162d61f4ee88b66ed6c3717f9e0: Loaded TSK: 0
I20260619 14:34:34.896883 14879 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 14:34:34.995097 14868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:34:34.995462 14868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:34:34.995527 14868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:34:35.001307 14868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:34:35.001425 14868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:34:35.001539 14868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.1
I20260619 14:34:35.009991 14868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:36237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.2.20:59627/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab
--principal=kudu/127.2.2.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.1:32785
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.2.1
--webserver_port=35945
--tserver_master_addrs=127.2.2.62:42457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.1
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:34:35.011713 14868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:34:35.013156 14868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:34:35.027251 14904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:34:35.027320 14903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:34:35.027900 14906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:34:35.028712 14868 server_base.cc:1034] running on GCE node
I20260619 14:34:35.029277 14868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:34:35.030195 14868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:34:35.037631 14868 hybrid_clock.cc:648] HybridClock initialized: now 1781879675037590 us; error 48 us; skew 500 ppm
Jun 19 14:34:35 dist-test-slave-70rm krb5kdc[13834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879675, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:34:35.041916 14868 init.cc:377] Logged in from keytab as kudu/127.2.2.1@KRBTEST.COM (short username kudu)
I20260619 14:34:35.043256 14868 webserver.cc:460] Webserver started at http://127.2.2.1:35945/ using document root <none> and password file <none>
I20260619 14:34:35.043973 14868 fs_manager.cc:359] Metadata directory not provided
I20260619 14:34:35.044075 14868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 14:34:35 dist-test-slave-70rm krb5kdc[13834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879675, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.1@KRBTEST.COM for HTTP/127.2.2.20@KRBTEST.COM
I20260619 14:34:35.069880 14868 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 14:34:35.073256 14914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:34:35.075031 14868 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260619 14:34:35.075168 14868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-0/wal
uuid: "d4f0800abcef4b468e118c96c42a3c2d"
format_stamp: "Formatted at 2026-06-19 14:34:33 on dist-test-slave-70rm"
server_key: "0a1fa5a1c6e609b1b4a832efc04babd5"
server_key_iv: "6364b2586031276b9c2b51fd449d955e"
server_key_version: "kuduclusterkey@0"
I20260619 14:34:35.075891 14868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:34:35.094326 14868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:34:35.098140 14868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:34:35.098356 14868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:34:35.099121 14868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:34:35.100224 14868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:34:35.100291 14868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:34:35.100347 14868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:34:35.100426 14868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260619 14:34:35.113253 14868 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.1:32785
I20260619 14:34:35.114724 14868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
I20260619 14:34:35.116953 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 14868
I20260619 14:34:35.117455 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.2.20:59627/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.2:33091
--local_ip_for_outbound_sockets=127.2.2.2
--tserver_master_addrs=127.2.2.62:42457
--webserver_port=43233
--webserver_interface=127.2.2.2
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab
--principal=kudu/127.2.2.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:36237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.2.20:59627/kms with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:34:35.129483 15027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.1:32785 every 8 connection(s)
Jun 19 14:34:35 dist-test-slave-70rm krb5kdc[13834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879675, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.1@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:34:35.170655 15031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:35.132824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:52683 (local address 127.2.2.62:42457)
0619 14:34:35.142490 (+ 9666us) server_negotiation.cc:206] Beginning negotiation
0619 14:34:35.142517 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0619 14:34:35.143068 (+ 551us) server_negotiation.cc:407] Connection header received
0619 14:34:35.144462 (+ 1394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:34:35.144473 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:34:35.144855 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:34:35.145237 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:34:35.147012 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:35.148540 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:35.150283 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:35.151002 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:35.151157 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:35.155686 (+ 4529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:34:35.155693 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:34:35.155710 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:34:35.155746 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:34:35.162498 (+ 6752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:35.164377 (+ 1879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:35.164384 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:35.164391 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:35.164567 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:35.168149 (+ 3582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:35.168153 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:35.168155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:35.168588 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:34:35.168757 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 14:34:35.169458 (+ 701us) server_negotiation.cc:299] Negotiation successful
0619 14:34:35.169753 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9443,"thread_start_us":144,"threads_started":1}
I20260619 14:34:35.170665 15032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:35.133759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:42457 (local address 127.2.2.1:52683)
0619 14:34:35.142857 (+ 9098us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:35.142914 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 14:34:35.143982 (+ 1068us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:34:35.145784 (+ 1802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:34:35.145794 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:34:35.146190 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:34:35.146816 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:35.146835 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:35.148717 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:35.148721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:35.150138 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:35.150151 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:35.151316 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:35.151374 (+ 58us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:35.151555 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:35.153085 (+ 1530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:34:35.153299 (+ 214us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:34:35.155425 (+ 2126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:34:35.162711 (+ 7286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:34:35.162717 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:34:35.162731 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:34:35.164199 (+ 1468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:34:35.165017 (+ 818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:34:35.165021 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:34:35.165023 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:34:35.165222 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:34:35.168735 (+ 3513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:34:35.168742 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:34:35.169059 (+ 317us) client_negotiation.cc:769] Sending connection context
0619 14:34:35.169316 (+ 257us) client_negotiation.cc:240] Negotiation successful
0619 14:34:35.169664 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8864,"thread_start_us":171,"threads_started":1}
I20260619 14:34:35.173658 15028 heartbeater.cc:347] Connected to a master server at 127.2.2.62:42457
I20260619 14:34:35.173904 15028 heartbeater.cc:464] Registering TS with master...
I20260619 14:34:35.174530 15028 heartbeater.cc:511] Master 127.2.2.62:42457 requested a full tablet report, sending...
W20260619 14:34:35.176699 15028 heartbeater.cc:644] Failed to heartbeat to 127.2.2.62:42457 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 14:34:35.176841 15028 heartbeater.cc:464] Registering TS with master...
I20260619 14:34:35.177014 15028 heartbeater.cc:511] Master 127.2.2.62:42457 requested a full tablet report, sending...
I20260619 14:34:35.177855 15028 heartbeater.cc:464] Registering TS with master...
I20260619 14:34:35.178056 15028 heartbeater.cc:511] Master 127.2.2.62:42457 requested a full tablet report, sending...
W20260619 14:34:35.178879 15028 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 14:34:35.283722 15030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:34:35.284125 15030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:34:35.284260 15030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:34:35.290796 15030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:34:35.290894 15030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:34:35.290992 15030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.2
I20260619 14:34:35.299683 15030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:36237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.2.20:59627/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab
--principal=kudu/127.2.2.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.2:33091
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.2.2
--webserver_port=43233
--tserver_master_addrs=127.2.2.62:42457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.2
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:34:35.301249 15030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:34:35.302790 15030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:34:35.315865 15037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:34:35.315876 15038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:34:35.316154 15030 server_base.cc:1034] running on GCE node
W20260619 14:34:35.315882 15040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:34:35.317165 15030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:34:35.317847 15030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:34:35.319090 15030 hybrid_clock.cc:648] HybridClock initialized: now 1781879675319052 us; error 60 us; skew 500 ppm
Jun 19 14:34:35 dist-test-slave-70rm krb5kdc[13834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879675, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:34:35.322301 15030 init.cc:377] Logged in from keytab as kudu/127.2.2.2@KRBTEST.COM (short username kudu)
I20260619 14:34:35.323326 15030 webserver.cc:460] Webserver started at http://127.2.2.2:43233/ using document root <none> and password file <none>
I20260619 14:34:35.323896 15030 fs_manager.cc:359] Metadata directory not provided
I20260619 14:34:35.323972 15030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 14:34:35 dist-test-slave-70rm krb5kdc[13834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879675, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.2@KRBTEST.COM for HTTP/127.2.2.20@KRBTEST.COM
I20260619 14:34:35.359469 15030 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260619 14:34:35.362443 15048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:34:35.364022 15030 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 14:34:35.364153 15030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-1/wal
uuid: "bdddfb4b708d4441b93ae829351f92e1"
format_stamp: "Formatted at 2026-06-19 14:34:33 on dist-test-slave-70rm"
server_key: "1c8264203e9ec933410a810eddcd8613"
server_key_iv: "c401c763d1f3378b18803189880aeafa"
server_key_version: "kuduclusterkey@0"
I20260619 14:34:35.364737 15030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:34:35.396901 15030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:34:35.400280 15030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:34:35.400503 15030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:34:35.401178 15030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:34:35.402096 15030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:34:35.402220 15030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:34:35.402318 15030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:34:35.402371 15030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:34:35.414093 15030 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.2:33091
I20260619 14:34:35.415998 15030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
I20260619 14:34:35.418480 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 15030
I20260619 14:34:35.419010 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.2.20:59627/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.3:39967
--local_ip_for_outbound_sockets=127.2.2.3
--tserver_master_addrs=127.2.2.62:42457
--webserver_port=41579
--webserver_interface=127.2.2.3
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab
--principal=kudu/127.2.2.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:36237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.2.20:59627/kms with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:34:35.416965 15162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.2:33091 every 8 connection(s)
Jun 19 14:34:35 dist-test-slave-70rm krb5kdc[13834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879675, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.2@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:34:35.453778 15031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:35.422283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:34739 (local address 127.2.2.62:42457)
0619 14:34:35.422446 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 14:34:35.422450 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:34:35.430027 (+ 7577us) server_negotiation.cc:407] Connection header received
0619 14:34:35.436075 (+ 6048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:34:35.436080 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:34:35.436162 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:34:35.436233 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:34:35.438309 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:35.439180 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:35.441101 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:35.441450 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:35.441541 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:35.449111 (+ 7570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:34:35.449118 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:34:35.449121 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:34:35.449182 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:34:35.451142 (+ 1960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:35.451889 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:35.451895 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:35.451897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:35.451993 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:35.452408 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:35.452412 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:35.452413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:35.452669 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:34:35.452754 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 14:34:35.453471 (+ 717us) server_negotiation.cc:299] Negotiation successful
0619 14:34:35.453614 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 14:34:35.455374 15166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:35.421052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:42457 (local address 127.2.2.2:34739)
0619 14:34:35.429795 (+ 8743us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:35.429877 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0619 14:34:35.435671 (+ 5794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:34:35.436906 (+ 1235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:34:35.436919 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:34:35.437312 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:34:35.438065 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:35.438086 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:35.439368 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:35.439372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:35.440929 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:35.440942 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:35.441698 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:35.441772 (+ 74us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:35.441991 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:35.443875 (+ 1884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:34:35.443899 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:34:35.448796 (+ 4897us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:34:35.451318 (+ 2522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:34:35.451326 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:34:35.451347 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:34:35.451752 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:34:35.452107 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:34:35.452110 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:34:35.452112 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:34:35.452290 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:34:35.452833 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:34:35.452841 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:34:35.453180 (+ 339us) client_negotiation.cc:769] Sending connection context
0619 14:34:35.454035 (+ 855us) client_negotiation.cc:240] Negotiation successful
0619 14:34:35.454411 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8386,"thread_start_us":176,"threads_started":1}
I20260619 14:34:35.456780 15163 heartbeater.cc:347] Connected to a master server at 127.2.2.62:42457
I20260619 14:34:35.457016 15163 heartbeater.cc:464] Registering TS with master...
I20260619 14:34:35.457572 15163 heartbeater.cc:511] Master 127.2.2.62:42457 requested a full tablet report, sending...
W20260619 14:34:35.459419 15163 heartbeater.cc:644] Failed to heartbeat to 127.2.2.62:42457 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 14:34:35.459563 15163 heartbeater.cc:464] Registering TS with master...
I20260619 14:34:35.459753 15163 heartbeater.cc:511] Master 127.2.2.62:42457 requested a full tablet report, sending...
I20260619 14:34:35.460772 15163 heartbeater.cc:464] Registering TS with master...
I20260619 14:34:35.460974 15163 heartbeater.cc:511] Master 127.2.2.62:42457 requested a full tablet report, sending...
W20260619 14:34:35.461875 15163 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 14:34:35.575326 15165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:34:35.575634 15165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:34:35.575743 15165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:34:35.580390 15165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:34:35.580555 15165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:34:35.580695 15165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.3
I20260619 14:34:35.588912 15165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:36237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.2.20:59627/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab
--principal=kudu/127.2.2.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.3:39967
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.2.3
--webserver_port=41579
--tserver_master_addrs=127.2.2.62:42457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.3
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:34:35.590660 15165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:34:35.592059 15165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:34:35.602912 15172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:34:35.602921 15171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:34:35.602921 15174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:34:35.603485 15165 server_base.cc:1034] running on GCE node
I20260619 14:34:35.603915 15165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:34:35.604725 15165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:34:35.605970 15165 hybrid_clock.cc:648] HybridClock initialized: now 1781879675605945 us; error 44 us; skew 500 ppm
Jun 19 14:34:35 dist-test-slave-70rm krb5kdc[13834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879675, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:34:35.609675 15165 init.cc:377] Logged in from keytab as kudu/127.2.2.3@KRBTEST.COM (short username kudu)
I20260619 14:34:35.610819 15165 webserver.cc:460] Webserver started at http://127.2.2.3:41579/ using document root <none> and password file <none>
I20260619 14:34:35.611491 15165 fs_manager.cc:359] Metadata directory not provided
I20260619 14:34:35.611581 15165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 14:34:35 dist-test-slave-70rm krb5kdc[13834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879675, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.3@KRBTEST.COM for HTTP/127.2.2.20@KRBTEST.COM
I20260619 14:34:35.639124 15165 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 14:34:35.642344 15182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:34:35.643790 15165 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:34:35.643975 15165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-2/wal
uuid: "bd5b0d612d214923b64c57044de4b6d0"
format_stamp: "Formatted at 2026-06-19 14:34:34 on dist-test-slave-70rm"
server_key: "db9598549446f5d279d7968288f4a7dc"
server_key_iv: "333a7f583fc4a8ebac94cbf81cac0dab"
server_key_version: "kuduclusterkey@0"
I20260619 14:34:35.644613 15165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:34:35.661072 15165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:34:35.664812 15165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:34:35.665074 15165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:34:35.665812 15165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:34:35.666956 15165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:34:35.667070 15165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:34:35.667176 15165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:34:35.667246 15165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:34:35.679086 15165 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.3:39967
I20260619 14:34:35.679107 15295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.3:39967 every 8 connection(s)
I20260619 14:34:35.680377 15165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
I20260619 14:34:35.687152 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 15165
Jun 19 14:34:35 dist-test-slave-70rm krb5kdc[13834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879675, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.3@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:34:35.700166 15300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:35.687928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38328 (local address 127.2.2.62:42457)
0619 14:34:35.688203 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0619 14:34:35.688208 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:34:35.688473 (+ 265us) server_negotiation.cc:407] Connection header received
0619 14:34:35.688670 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:34:35.688673 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:34:35.688723 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:34:35.688786 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:34:35.689826 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:35.690632 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:35.691848 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:35.692188 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:35.692300 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:35.694286 (+ 1986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:34:35.694290 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:34:35.694293 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:34:35.694324 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:34:35.696191 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:35.696832 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:35.696835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:35.696837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:35.697263 (+ 426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:35.698263 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:35.698266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:35.698268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:35.698495 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:34:35.698720 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0619 14:34:35.699844 (+ 1124us) server_negotiation.cc:299] Negotiation successful
0619 14:34:35.699972 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":91,"threads_started":1}
I20260619 14:34:35.701920 15031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:35.684086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.3:52941 (local address 127.2.2.62:42457)
0619 14:34:35.684218 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 14:34:35.684221 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:34:35.684237 (+ 16us) server_negotiation.cc:407] Connection header received
0619 14:34:35.685052 (+ 815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:34:35.685056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:34:35.685112 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:34:35.685203 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:34:35.686724 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:35.687301 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:35.688970 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:35.689348 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:35.689428 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:35.696372 (+ 6944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:34:35.696376 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:34:35.696380 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:34:35.696405 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:34:35.698152 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:35.699306 (+ 1154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:35.699310 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:35.699312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:35.699391 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:35.700625 (+ 1234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:35.700628 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:35.700630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:35.700830 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:34:35.700898 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 14:34:35.701654 (+ 756us) server_negotiation.cc:299] Negotiation successful
0619 14:34:35.701772 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 14:34:35.703585 15298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:35.683028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:42457 (local address 127.2.2.3:52941)
0619 14:34:35.683624 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:35.683688 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 14:34:35.684768 (+ 1080us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:34:35.685423 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:34:35.685433 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:34:35.685800 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:34:35.686538 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:35.686560 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:35.687432 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:35.687435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:35.688821 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:35.688833 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:35.689588 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:35.689647 (+ 59us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:35.689852 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:35.690567 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:34:35.690588 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:34:35.694074 (+ 3486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:34:35.698787 (+ 4713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:34:35.698794 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:34:35.698813 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:34:35.699187 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:34:35.699512 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:34:35.699516 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:34:35.699518 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:34:35.699701 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:34:35.700980 (+ 1279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:34:35.700987 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:34:35.701343 (+ 356us) client_negotiation.cc:769] Sending connection context
0619 14:34:35.702138 (+ 795us) client_negotiation.cc:240] Negotiation successful
0619 14:34:35.702519 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":118,"threads_started":1}
I20260619 14:34:35.704821 15296 heartbeater.cc:347] Connected to a master server at 127.2.2.62:42457
I20260619 14:34:35.705121 15296 heartbeater.cc:464] Registering TS with master...
I20260619 14:34:35.705719 15296 heartbeater.cc:511] Master 127.2.2.62:42457 requested a full tablet report, sending...
I20260619 14:34:35.707234 14830 ts_manager.cc:194] Registered new tserver with Master: bd5b0d612d214923b64c57044de4b6d0 (127.2.2.3:39967)
I20260619 14:34:35.708814 14830 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.2.3@KRBTEST.COM'} at 127.2.2.3:52941
Jun 19 14:34:36 dist-test-slave-70rm krb5kdc[13834](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781879676, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:34:36.179142 15028 heartbeater.cc:464] Registering TS with master...
I20260619 14:34:36.179469 15028 heartbeater.cc:511] Master 127.2.2.62:42457 requested a full tablet report, sending...
I20260619 14:34:36.180321 14830 ts_manager.cc:194] Registered new tserver with Master: d4f0800abcef4b468e118c96c42a3c2d (127.2.2.1:32785)
I20260619 14:34:36.181221 14830 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.2.1@KRBTEST.COM'} at 127.2.2.1:52683
Jun 19 14:34:36 dist-test-slave-70rm krb5kdc[13834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879676, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.62@KRBTEST.COM for HTTP/127.2.2.20@KRBTEST.COM
I20260619 14:34:36.462101 15163 heartbeater.cc:464] Registering TS with master...
I20260619 14:34:36.462458 15163 heartbeater.cc:511] Master 127.2.2.62:42457 requested a full tablet report, sending...
I20260619 14:34:36.463289 14830 ts_manager.cc:194] Registered new tserver with Master: bdddfb4b708d4441b93ae829351f92e1 (127.2.2.2:33091)
I20260619 14:34:36.464040 14830 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.2.2@KRBTEST.COM'} at 127.2.2.2:34739
I20260619 14:34:36.478945 2056 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:34:36.490427 15318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:36.480838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43714 (local address 127.2.2.62:42457)
0619 14:34:36.481123 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0619 14:34:36.481128 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:34:36.481249 (+ 121us) server_negotiation.cc:407] Connection header received
0619 14:34:36.481389 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:34:36.481392 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:34:36.481452 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:34:36.481594 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:34:36.482415 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:36.483384 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:36.484567 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:36.484926 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:36.485051 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:36.486673 (+ 1622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:34:36.486676 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:34:36.486680 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:34:36.486707 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:34:36.488571 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:36.489045 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:36.489049 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:36.489051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:36.489143 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:36.489446 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:36.489450 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:36.489452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:36.489686 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:34:36.489826 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 14:34:36.490074 (+ 248us) server_negotiation.cc:299] Negotiation successful
0619 14:34:36.490247 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":91,"threads_started":1}
I20260619 14:34:36.494915 14830 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43714:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:34:36.499907 14830 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:34:36.519794 15323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:36.511827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.2:33091 (local address 127.0.0.1:60732)
0619 14:34:36.513806 (+ 1979us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:36.513855 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 14:34:36.514035 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:34:36.514773 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:34:36.514779 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:34:36.514799 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:34:36.515101 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:36.515107 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:36.516730 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:36.516734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:36.518042 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:36.518053 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:36.519271 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:36.519275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:36.519376 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:36.519411 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 14:34:36.519519 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 14:34:36.519636 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1822,"mutex_wait_us":184,"thread_start_us":96,"threads_started":1}
I20260619 14:34:36.520387 15326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:36.512590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60732 (local address 127.2.2.2:33091)
0619 14:34:36.513398 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0619 14:34:36.513406 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:34:36.513935 (+ 529us) server_negotiation.cc:407] Connection header received
0619 14:34:36.514242 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:34:36.514248 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:34:36.514469 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:34:36.514585 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:34:36.515181 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:36.516597 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:36.518198 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:36.518971 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:36.519046 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:36.519185 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 14:34:36.520158 (+ 973us) server_negotiation.cc:299] Negotiation successful
0619 14:34:36.520263 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":700,"thread_start_us":60,"threads_started":1}
I20260619 14:34:36.520790 15327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:36.511833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.3:39967 (local address 127.0.0.1:44634)
0619 14:34:36.513980 (+ 2147us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:36.513995 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:34:36.514091 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:34:36.514789 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:34:36.514792 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:34:36.514808 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:34:36.514961 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:36.514981 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:36.517015 (+ 2034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:36.517019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:36.518224 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:36.518235 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:36.520385 (+ 2150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:36.520389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:36.520480 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:36.520502 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 14:34:36.520573 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 14:34:36.520659 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2070,"mutex_wait_us":3,"thread_start_us":119,"threads_started":1}
I20260619 14:34:36.521353 15325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:36.511967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44634 (local address 127.2.2.3:39967)
0619 14:34:36.513584 (+ 1617us) server_negotiation.cc:206] Beginning negotiation
0619 14:34:36.513589 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:34:36.514229 (+ 640us) server_negotiation.cc:407] Connection header received
0619 14:34:36.514313 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:34:36.514318 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:34:36.514511 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:34:36.514650 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:34:36.515221 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:36.516870 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:36.519281 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:36.520092 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:36.520188 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:36.520321 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 14:34:36.521126 (+ 805us) server_negotiation.cc:299] Negotiation successful
0619 14:34:36.521224 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1531,"thread_start_us":72,"threads_started":1}
I20260619 14:34:36.524106 15080 tablet_service.cc:1467] Processing CreateTablet for tablet d7003154cfe44307a6c8a5d3f78a4fd2 (DEFAULT_TABLE table=test-table [id=1dcbb46a07514a53bac2246332ea3384]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:34:36.525274 15080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7003154cfe44307a6c8a5d3f78a4fd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:34:36.525831 15324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:36.513237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.1:32785 (local address 127.0.0.1:46194)
0619 14:34:36.514992 (+ 1755us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:36.515003 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:34:36.515073 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:34:36.520848 (+ 5775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:34:36.520852 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:34:36.520874 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:34:36.520971 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:36.520977 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:36.522436 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:36.522439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:36.523543 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:36.523554 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:36.525427 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:36.525430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:36.525532 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:36.525556 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 14:34:36.525626 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 14:34:36.525697 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1702,"thread_start_us":75,"threads_started":1}
I20260619 14:34:36.526348 15328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:36.513590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46194 (local address 127.2.2.1:32785)
0619 14:34:36.520184 (+ 6594us) server_negotiation.cc:206] Beginning negotiation
0619 14:34:36.520191 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:34:36.520215 (+ 24us) server_negotiation.cc:407] Connection header received
0619 14:34:36.520292 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:34:36.520297 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:34:36.520541 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:34:36.520691 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:34:36.521109 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:36.522299 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:36.523683 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:36.525124 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:36.525227 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:36.525362 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 14:34:36.526085 (+ 723us) server_negotiation.cc:299] Negotiation successful
0619 14:34:36.526215 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6464,"thread_start_us":112,"threads_started":1}
I20260619 14:34:36.526546 15230 tablet_service.cc:1467] Processing CreateTablet for tablet d7003154cfe44307a6c8a5d3f78a4fd2 (DEFAULT_TABLE table=test-table [id=1dcbb46a07514a53bac2246332ea3384]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:34:36.527607 15230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7003154cfe44307a6c8a5d3f78a4fd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:34:36.529011 14962 tablet_service.cc:1467] Processing CreateTablet for tablet d7003154cfe44307a6c8a5d3f78a4fd2 (DEFAULT_TABLE table=test-table [id=1dcbb46a07514a53bac2246332ea3384]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:34:36.530077 14962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7003154cfe44307a6c8a5d3f78a4fd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:34:36.536324 15329 tablet_bootstrap.cc:492] T d7003154cfe44307a6c8a5d3f78a4fd2 P bd5b0d612d214923b64c57044de4b6d0: Bootstrap starting.
I20260619 14:34:36.537436 15330 tablet_bootstrap.cc:492] T d7003154cfe44307a6c8a5d3f78a4fd2 P bdddfb4b708d4441b93ae829351f92e1: Bootstrap starting.
I20260619 14:34:36.538830 15329 tablet_bootstrap.cc:654] T d7003154cfe44307a6c8a5d3f78a4fd2 P bd5b0d612d214923b64c57044de4b6d0: Neither blocks nor log segments found. Creating new log.
I20260619 14:34:36.539750 15329 log.cc:824] T d7003154cfe44307a6c8a5d3f78a4fd2 P bd5b0d612d214923b64c57044de4b6d0: Log is configured to *not* fsync() on all Append() calls
I20260619 14:34:36.539897 15331 tablet_bootstrap.cc:492] T d7003154cfe44307a6c8a5d3f78a4fd2 P d4f0800abcef4b468e118c96c42a3c2d: Bootstrap starting.
I20260619 14:34:36.541028 15330 tablet_bootstrap.cc:654] T d7003154cfe44307a6c8a5d3f78a4fd2 P bdddfb4b708d4441b93ae829351f92e1: Neither blocks nor log segments found. Creating new log.
I20260619 14:34:36.541947 15331 tablet_bootstrap.cc:654] T d7003154cfe44307a6c8a5d3f78a4fd2 P d4f0800abcef4b468e118c96c42a3c2d: Neither blocks nor log segments found. Creating new log.
I20260619 14:34:36.542033 15329 tablet_bootstrap.cc:492] T d7003154cfe44307a6c8a5d3f78a4fd2 P bd5b0d612d214923b64c57044de4b6d0: No bootstrap required, opened a new log
I20260619 14:34:36.541936 15330 log.cc:824] T d7003154cfe44307a6c8a5d3f78a4fd2 P bdddfb4b708d4441b93ae829351f92e1: Log is configured to *not* fsync() on all Append() calls
I20260619 14:34:36.542279 15329 ts_tablet_manager.cc:1397] T d7003154cfe44307a6c8a5d3f78a4fd2 P bd5b0d612d214923b64c57044de4b6d0: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260619 14:34:36.542802 15331 log.cc:824] T d7003154cfe44307a6c8a5d3f78a4fd2 P d4f0800abcef4b468e118c96c42a3c2d: Log is configured to *not* fsync() on all Append() calls
I20260619 14:34:36.544313 15330 tablet_bootstrap.cc:492] T d7003154cfe44307a6c8a5d3f78a4fd2 P bdddfb4b708d4441b93ae829351f92e1: No bootstrap required, opened a new log
I20260619 14:34:36.544505 15330 ts_tablet_manager.cc:1397] T d7003154cfe44307a6c8a5d3f78a4fd2 P bdddfb4b708d4441b93ae829351f92e1: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 14:34:36.544727 15331 tablet_bootstrap.cc:492] T d7003154cfe44307a6c8a5d3f78a4fd2 P d4f0800abcef4b468e118c96c42a3c2d: No bootstrap required, opened a new log
I20260619 14:34:36.544914 15331 ts_tablet_manager.cc:1397] T d7003154cfe44307a6c8a5d3f78a4fd2 P d4f0800abcef4b468e118c96c42a3c2d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 14:34:36.546684 15329 raft_consensus.cc:348] T d7003154cfe44307a6c8a5d3f78a4fd2 P bd5b0d612d214923b64c57044de4b6d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd5b0d612d214923b64c57044de4b6d0" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 39967 } } peers { permanent_uuid: "bdddfb4b708d4441b93ae829351f92e1" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 33091 } } peers { permanent_uuid: "d4f0800abcef4b468e118c96c42a3c2d" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 32785 } }
I20260619 14:34:36.546936 15329 raft_consensus.cc:374] T d7003154cfe44307a6c8a5d3f78a4fd2 P bd5b0d612d214923b64c57044de4b6d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:34:36.546978 15329 raft_consensus.cc:729] T d7003154cfe44307a6c8a5d3f78a4fd2 P bd5b0d612d214923b64c57044de4b6d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd5b0d612d214923b64c57044de4b6d0, State: Initialized, Role: FOLLOWER
I20260619 14:34:36.547407 15329 consensus_queue.cc:260] T d7003154cfe44307a6c8a5d3f78a4fd2 P bd5b0d612d214923b64c57044de4b6d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd5b0d612d214923b64c57044de4b6d0" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 39967 } } peers { permanent_uuid: "bdddfb4b708d4441b93ae829351f92e1" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 33091 } } peers { permanent_uuid: "d4f0800abcef4b468e118c96c42a3c2d" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 32785 } }
I20260619 14:34:36.548437 15329 ts_tablet_manager.cc:1428] T d7003154cfe44307a6c8a5d3f78a4fd2 P bd5b0d612d214923b64c57044de4b6d0: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 14:34:36.548827 15296 heartbeater.cc:503] Master 127.2.2.62:42457 was elected leader, sending a full tablet report...
I20260619 14:34:36.548913 15331 raft_consensus.cc:348] T d7003154cfe44307a6c8a5d3f78a4fd2 P d4f0800abcef4b468e118c96c42a3c2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd5b0d612d214923b64c57044de4b6d0" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 39967 } } peers { permanent_uuid: "bdddfb4b708d4441b93ae829351f92e1" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 33091 } } peers { permanent_uuid: "d4f0800abcef4b468e118c96c42a3c2d" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 32785 } }
I20260619 14:34:36.549160 15331 raft_consensus.cc:374] T d7003154cfe44307a6c8a5d3f78a4fd2 P d4f0800abcef4b468e118c96c42a3c2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:34:36.549245 15331 raft_consensus.cc:729] T d7003154cfe44307a6c8a5d3f78a4fd2 P d4f0800abcef4b468e118c96c42a3c2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4f0800abcef4b468e118c96c42a3c2d, State: Initialized, Role: FOLLOWER
I20260619 14:34:36.549789 15331 consensus_queue.cc:260] T d7003154cfe44307a6c8a5d3f78a4fd2 P d4f0800abcef4b468e118c96c42a3c2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd5b0d612d214923b64c57044de4b6d0" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 39967 } } peers { permanent_uuid: "bdddfb4b708d4441b93ae829351f92e1" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 33091 } } peers { permanent_uuid: "d4f0800abcef4b468e118c96c42a3c2d" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 32785 } }
I20260619 14:34:36.550875 15331 ts_tablet_manager.cc:1428] T d7003154cfe44307a6c8a5d3f78a4fd2 P d4f0800abcef4b468e118c96c42a3c2d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 14:34:36.550981 15330 raft_consensus.cc:348] T d7003154cfe44307a6c8a5d3f78a4fd2 P bdddfb4b708d4441b93ae829351f92e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd5b0d612d214923b64c57044de4b6d0" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 39967 } } peers { permanent_uuid: "bdddfb4b708d4441b93ae829351f92e1" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 33091 } } peers { permanent_uuid: "d4f0800abcef4b468e118c96c42a3c2d" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 32785 } }
I20260619 14:34:36.551213 15028 heartbeater.cc:503] Master 127.2.2.62:42457 was elected leader, sending a full tablet report...
I20260619 14:34:36.551241 15330 raft_consensus.cc:374] T d7003154cfe44307a6c8a5d3f78a4fd2 P bdddfb4b708d4441b93ae829351f92e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:34:36.551290 15330 raft_consensus.cc:729] T d7003154cfe44307a6c8a5d3f78a4fd2 P bdddfb4b708d4441b93ae829351f92e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdddfb4b708d4441b93ae829351f92e1, State: Initialized, Role: FOLLOWER
I20260619 14:34:36.551762 15330 consensus_queue.cc:260] T d7003154cfe44307a6c8a5d3f78a4fd2 P bdddfb4b708d4441b93ae829351f92e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd5b0d612d214923b64c57044de4b6d0" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 39967 } } peers { permanent_uuid: "bdddfb4b708d4441b93ae829351f92e1" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 33091 } } peers { permanent_uuid: "d4f0800abcef4b468e118c96c42a3c2d" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 32785 } }
I20260619 14:34:36.553437 15330 ts_tablet_manager.cc:1428] T d7003154cfe44307a6c8a5d3f78a4fd2 P bdddfb4b708d4441b93ae829351f92e1: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260619 14:34:36.553889 15163 heartbeater.cc:503] Master 127.2.2.62:42457 was elected leader, sending a full tablet report...
W20260619 14:34:36.631832 15029 tablet.cc:2369] T d7003154cfe44307a6c8a5d3f78a4fd2 P d4f0800abcef4b468e118c96c42a3c2d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:34:36.669899 15164 tablet.cc:2369] T d7003154cfe44307a6c8a5d3f78a4fd2 P bdddfb4b708d4441b93ae829351f92e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:34:36.682111 15297 tablet.cc:2369] T d7003154cfe44307a6c8a5d3f78a4fd2 P bd5b0d612d214923b64c57044de4b6d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 14:34:36 dist-test-slave-70rm krb5kdc[13834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879676, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.62@KRBTEST.COM for HTTP/127.2.2.20@KRBTEST.COM
I20260619 14:34:36.778965 15336 raft_consensus.cc:482] T d7003154cfe44307a6c8a5d3f78a4fd2 P d4f0800abcef4b468e118c96c42a3c2d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:34:36.779222 15336 raft_consensus.cc:504] T d7003154cfe44307a6c8a5d3f78a4fd2 P d4f0800abcef4b468e118c96c42a3c2d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd5b0d612d214923b64c57044de4b6d0" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 39967 } } peers { permanent_uuid: "bdddfb4b708d4441b93ae829351f92e1" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 33091 } } peers { permanent_uuid: "d4f0800abcef4b468e118c96c42a3c2d" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 32785 } }
I20260619 14:34:36.780681 15336 leader_election.cc:288] T d7003154cfe44307a6c8a5d3f78a4fd2 P d4f0800abcef4b468e118c96c42a3c2d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd5b0d612d214923b64c57044de4b6d0 (127.2.2.3:39967), bdddfb4b708d4441b93ae829351f92e1 (127.2.2.2:33091)
I20260619 14:34:36.787535 15325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:36.781937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:57267 (local address 127.2.2.3:39967)
0619 14:34:36.782146 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 14:34:36.782153 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:34:36.782713 (+ 560us) server_negotiation.cc:407] Connection header received
0619 14:34:36.782937 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:34:36.782941 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:34:36.782999 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:34:36.783053 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:34:36.783616 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:36.784412 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:36.786133 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:36.786743 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:36.786833 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:36.786908 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 14:34:36.787289 (+ 381us) server_negotiation.cc:299] Negotiation successful
0619 14:34:36.787379 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260619 14:34:36.788048 15341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:36.781084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.3:39967 (local address 127.2.2.1:57267)
0619 14:34:36.782605 (+ 1521us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:36.782622 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:34:36.782803 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:34:36.783279 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:34:36.783282 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:34:36.783307 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:34:36.783458 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:36.783466 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:36.784600 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:36.784604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:36.785943 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:36.785955 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:36.787012 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:36.787016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:36.787127 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:36.787153 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 14:34:36.787795 (+ 642us) client_negotiation.cc:240] Negotiation successful
0619 14:34:36.787886 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1447,"thread_start_us":135,"threads_started":1}
I20260619 14:34:36.788751 15250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7003154cfe44307a6c8a5d3f78a4fd2" candidate_uuid: "d4f0800abcef4b468e118c96c42a3c2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd5b0d612d214923b64c57044de4b6d0" is_pre_election: true
I20260619 14:34:36.789119 15250 raft_consensus.cc:2454] T d7003154cfe44307a6c8a5d3f78a4fd2 P bd5b0d612d214923b64c57044de4b6d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4f0800abcef4b468e118c96c42a3c2d in term 0.
I20260619 14:34:36.789726 14917 leader_election.cc:302] T d7003154cfe44307a6c8a5d3f78a4fd2 P d4f0800abcef4b468e118c96c42a3c2d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd5b0d612d214923b64c57044de4b6d0, d4f0800abcef4b468e118c96c42a3c2d; no voters:
I20260619 14:34:36.790088 15336 raft_consensus.cc:2781] T d7003154cfe44307a6c8a5d3f78a4fd2 P d4f0800abcef4b468e118c96c42a3c2d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:34:36.790205 15336 raft_consensus.cc:482] T d7003154cfe44307a6c8a5d3f78a4fd2 P d4f0800abcef4b468e118c96c42a3c2d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:34:36.790258 15336 raft_consensus.cc:3037] T d7003154cfe44307a6c8a5d3f78a4fd2 P d4f0800abcef4b468e118c96c42a3c2d [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:34:36.791719 15336 raft_consensus.cc:504] T d7003154cfe44307a6c8a5d3f78a4fd2 P d4f0800abcef4b468e118c96c42a3c2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd5b0d612d214923b64c57044de4b6d0" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 39967 } } peers { permanent_uuid: "bdddfb4b708d4441b93ae829351f92e1" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 33091 } } peers { permanent_uuid: "d4f0800abcef4b468e118c96c42a3c2d" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 32785 } }
I20260619 14:34:36.792511 15250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7003154cfe44307a6c8a5d3f78a4fd2" candidate_uuid: "d4f0800abcef4b468e118c96c42a3c2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd5b0d612d214923b64c57044de4b6d0"
I20260619 14:34:36.792649 15250 raft_consensus.cc:3037] T d7003154cfe44307a6c8a5d3f78a4fd2 P bd5b0d612d214923b64c57044de4b6d0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:34:36.793972 15250 raft_consensus.cc:2454] T d7003154cfe44307a6c8a5d3f78a4fd2 P bd5b0d612d214923b64c57044de4b6d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4f0800abcef4b468e118c96c42a3c2d in term 1.
I20260619 14:34:36.794344 14917 leader_election.cc:302] T d7003154cfe44307a6c8a5d3f78a4fd2 P d4f0800abcef4b468e118c96c42a3c2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd5b0d612d214923b64c57044de4b6d0, d4f0800abcef4b468e118c96c42a3c2d; no voters:
I20260619 14:34:36.794883 15336 leader_election.cc:288] T d7003154cfe44307a6c8a5d3f78a4fd2 P d4f0800abcef4b468e118c96c42a3c2d [CANDIDATE]: Term 1 election: Requested vote from peers bd5b0d612d214923b64c57044de4b6d0 (127.2.2.3:39967), bdddfb4b708d4441b93ae829351f92e1 (127.2.2.2:33091)
I20260619 14:34:36.794993 15336 raft_consensus.cc:2781] T d7003154cfe44307a6c8a5d3f78a4fd2 P d4f0800abcef4b468e118c96c42a3c2d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:34:36.795073 15336 raft_consensus.cc:686] T d7003154cfe44307a6c8a5d3f78a4fd2 P d4f0800abcef4b468e118c96c42a3c2d [term 1 LEADER]: Becoming Leader. State: Replica: d4f0800abcef4b468e118c96c42a3c2d, State: Running, Role: LEADER
I20260619 14:34:36.795449 15336 consensus_queue.cc:237] T d7003154cfe44307a6c8a5d3f78a4fd2 P d4f0800abcef4b468e118c96c42a3c2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd5b0d612d214923b64c57044de4b6d0" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 39967 } } peers { permanent_uuid: "bdddfb4b708d4441b93ae829351f92e1" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 33091 } } peers { permanent_uuid: "d4f0800abcef4b468e118c96c42a3c2d" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 32785 } }
I20260619 14:34:36.799216 14829 catalog_manager.cc:5495] T d7003154cfe44307a6c8a5d3f78a4fd2 P d4f0800abcef4b468e118c96c42a3c2d reported cstate change: term changed from 0 to 1, leader changed from <none> to d4f0800abcef4b468e118c96c42a3c2d (127.2.2.1). New cstate: current_term: 1 leader_uuid: "d4f0800abcef4b468e118c96c42a3c2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd5b0d612d214923b64c57044de4b6d0" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 39967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bdddfb4b708d4441b93ae829351f92e1" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 33091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4f0800abcef4b468e118c96c42a3c2d" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 32785 } health_report { overall_health: HEALTHY } } }
I20260619 14:34:36.803629 15326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:36.781998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:34549 (local address 127.2.2.2:33091)
0619 14:34:36.782139 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 14:34:36.782144 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:34:36.788654 (+ 6510us) server_negotiation.cc:407] Connection header received
0619 14:34:36.788876 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:34:36.788880 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:34:36.788940 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:34:36.789045 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:34:36.800404 (+ 11359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:36.801131 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:36.802537 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:36.803017 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:36.803181 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:36.803252 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 14:34:36.803425 (+ 173us) server_negotiation.cc:299] Negotiation successful
0619 14:34:36.803501 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 14:34:36.803630 15341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:36.781522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.2:33091 (local address 127.2.2.1:34549)
0619 14:34:36.788572 (+ 7050us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:36.788587 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:34:36.788737 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:34:36.789220 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:34:36.789223 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:34:36.789243 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:34:36.789320 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:36.789327 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:36.801312 (+ 11985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:36.801316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:36.802388 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:36.802399 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:36.803179 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:36.803183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:36.803270 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:36.803299 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 14:34:36.803397 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 14:34:36.803492 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6984,"thread_start_us":115,"threads_started":1}
I20260619 14:34:36.804493 15115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7003154cfe44307a6c8a5d3f78a4fd2" candidate_uuid: "d4f0800abcef4b468e118c96c42a3c2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdddfb4b708d4441b93ae829351f92e1" is_pre_election: true
I20260619 14:34:36.804502 15098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7003154cfe44307a6c8a5d3f78a4fd2" candidate_uuid: "d4f0800abcef4b468e118c96c42a3c2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdddfb4b708d4441b93ae829351f92e1"
I20260619 14:34:36.804826 15115 raft_consensus.cc:2454] T d7003154cfe44307a6c8a5d3f78a4fd2 P bdddfb4b708d4441b93ae829351f92e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4f0800abcef4b468e118c96c42a3c2d in term 0.
I20260619 14:34:36.824566 15328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:36.819213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46198 (local address 127.2.2.1:32785)
0619 14:34:36.819355 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 14:34:36.819359 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:34:36.819378 (+ 19us) server_negotiation.cc:407] Connection header received
0619 14:34:36.819478 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:34:36.819481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:34:36.819555 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:34:36.819678 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:34:36.820097 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:36.820924 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:36.822283 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:36.822647 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:36.822746 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:36.823195 (+ 449us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:34:36.823779 (+ 584us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:34:36.823901 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 14:34:36.824339 (+ 438us) server_negotiation.cc:299] Negotiation successful
0619 14:34:36.824404 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 14:34:36.838335 15098 raft_consensus.cc:3037] T d7003154cfe44307a6c8a5d3f78a4fd2 P bdddfb4b708d4441b93ae829351f92e1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:34:36.839946 15098 raft_consensus.cc:1261] T d7003154cfe44307a6c8a5d3f78a4fd2 P bdddfb4b708d4441b93ae829351f92e1 [term 1 FOLLOWER]: Refusing update from remote peer d4f0800abcef4b468e118c96c42a3c2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:34:36.842186 15342 consensus_queue.cc:1035] T d7003154cfe44307a6c8a5d3f78a4fd2 P d4f0800abcef4b468e118c96c42a3c2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdddfb4b708d4441b93ae829351f92e1" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 33091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:34:36.843138 15250 raft_consensus.cc:1261] T d7003154cfe44307a6c8a5d3f78a4fd2 P bd5b0d612d214923b64c57044de4b6d0 [term 1 FOLLOWER]: Refusing update from remote peer d4f0800abcef4b468e118c96c42a3c2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:34:36.848075 15342 consensus_queue.cc:1035] T d7003154cfe44307a6c8a5d3f78a4fd2 P d4f0800abcef4b468e118c96c42a3c2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd5b0d612d214923b64c57044de4b6d0" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 39967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:34:36.858816 15345 mvcc.cc:205] Tried to move back new op lower bound from 7298579156321898496 to 7298579156157886464. Current Snapshot: MvccSnapshot[applied={T|T < 7298579156321898496}]
I20260619 14:34:36.864806 15346 mvcc.cc:205] Tried to move back new op lower bound from 7298579156321898496 to 7298579156157886464. Current Snapshot: MvccSnapshot[applied={T|T < 7298579156321898496}]
I20260619 14:34:36.866580 15348 mvcc.cc:205] Tried to move back new op lower bound from 7298579156321898496 to 7298579156157886464. Current Snapshot: MvccSnapshot[applied={T|T < 7298579156321898496}]
I20260619 14:34:36.876626 14830 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43714:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:34:36.877020 14830 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43714:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:34:36.879969 14830 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 839a0162d61f4ee88b66ed6c3717f9e0: Sending DeleteTablet for 3 replicas of tablet d7003154cfe44307a6c8a5d3f78a4fd2
I20260619 14:34:36.881322 15080 tablet_service.cc:1514] Processing DeleteTablet for tablet d7003154cfe44307a6c8a5d3f78a4fd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:34:36 UTC) from {username='kudu', principal='kudu/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:60732
I20260619 14:34:36.881336 15230 tablet_service.cc:1514] Processing DeleteTablet for tablet d7003154cfe44307a6c8a5d3f78a4fd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:34:36 UTC) from {username='kudu', principal='kudu/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:44634
I20260619 14:34:36.882208 14962 tablet_service.cc:1514] Processing DeleteTablet for tablet d7003154cfe44307a6c8a5d3f78a4fd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:34:36 UTC) from {username='kudu', principal='kudu/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:46194
I20260619 14:34:36.882901 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 14868
I20260619 14:34:36.888283 15355 tablet_replica.cc:330] T d7003154cfe44307a6c8a5d3f78a4fd2 P bd5b0d612d214923b64c57044de4b6d0: stopping tablet replica
I20260619 14:34:36.888619 15355 raft_consensus.cc:2229] T d7003154cfe44307a6c8a5d3f78a4fd2 P bd5b0d612d214923b64c57044de4b6d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:34:36.888944 15355 raft_consensus.cc:2258] T d7003154cfe44307a6c8a5d3f78a4fd2 P bd5b0d612d214923b64c57044de4b6d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:34:36.890601 15355 ts_tablet_manager.cc:1905] T d7003154cfe44307a6c8a5d3f78a4fd2 P bd5b0d612d214923b64c57044de4b6d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:34:36.893237 15355 ts_tablet_manager.cc:1918] T d7003154cfe44307a6c8a5d3f78a4fd2 P bd5b0d612d214923b64c57044de4b6d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:34:36.893342 15355 log.cc:1196] T d7003154cfe44307a6c8a5d3f78a4fd2 P bd5b0d612d214923b64c57044de4b6d0: Deleting WAL directory at /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879514993456-2056-0/minicluster-data/ts-2/wal/wals/d7003154cfe44307a6c8a5d3f78a4fd2
I20260619 14:34:36.893662 15355 ts_tablet_manager.cc:1939] T d7003154cfe44307a6c8a5d3f78a4fd2 P bd5b0d612d214923b64c57044de4b6d0: Deleting consensus metadata
I20260619 14:34:36.895027 14816 catalog_manager.cc:4841] TS bd5b0d612d214923b64c57044de4b6d0 (127.2.2.3:39967): tablet d7003154cfe44307a6c8a5d3f78a4fd2 (table test-table [id=1dcbb46a07514a53bac2246332ea3384]) successfully deleted
I20260619 14:34:36.898451 15354 tablet_replica.cc:330] T d7003154cfe44307a6c8a5d3f78a4fd2 P bdddfb4b708d4441b93ae829351f92e1: stopping tablet replica
I20260619 14:34:36.898900 15354 raft_consensus.cc:2229] T d7003154cfe44307a6c8a5d3f78a4fd2 P bdddfb4b708d4441b93ae829351f92e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:34:36.899336 15354 raft_consensus.cc:2258] T d7003154cfe44307a6c8a5d3f78a4fd2 P bdddfb4b708d4441b93ae829351f92e1 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 14:34:36.901234 14814 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.2.1:32785 (error 108)
I20260619 14:34:36.901283 15354 ts_tablet_manager.cc:1905] T d7003154cfe44307a6c8a5d3f78a4fd2 P bdddfb4b708d4441b93ae829351f92e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:34:36.902303 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 15030
I20260619 14:34:36.910238 15324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:36.909861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.1:32785 (local address 127.0.0.1:46204)
0619 14:34:36.910051 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:36.910115 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.1:32785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
W20260619 14:34:36.910444 14814 catalog_manager.cc:4584] TS d4f0800abcef4b468e118c96c42a3c2d (127.2.2.1:32785): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d7003154cfe44307a6c8a5d3f78a4fd2: Network error: Client connection negotiation failed: client connection to 127.2.2.1:32785: connect: Connection refused (error 111)
W20260619 14:34:36.911470 15324 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:34:36.910906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.2:33091 (local address 127.0.0.1:60734)
0619 14:34:36.911045 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:36.911058 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:34:36.911142 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:34:36.911378 (+ 236us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.2:33091: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":73}
I20260619 14:34:36.912096 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 15165
I20260619 14:34:36.920797 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 14797
I20260619 14:34:36.928546 2056 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 14:34:37.274000 2056 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 14:34:37.274150 2056 mini_ranger.cc:66] Stopping Ranger...
I20260619 14:34:37.619139 2056 mini_ranger.cc:68] Stopped Ranger
2026-06-19 14:34:37.619 UTC [13864] LOG: received smart shutdown request
2026-06-19 14:34:37.623 UTC [13864] LOG: background worker "logical replication launcher" (PID 13873) exited with exit code 1
2026-06-19 14:34:37.623 UTC [13868] LOG: shutting down
2026-06-19 14:34:37.808 UTC [13864] LOG: database system is shut down
2026-06-19T14:34:37Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49627 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:34:37 dist-test-slave-70rm krb5kdc[15366](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:34:37 dist-test-slave-70rm krb5kdc[15366](info): set up 2 sockets
Jun 19 14:34:37 dist-test-slave-70rm krb5kdc[15366](info): commencing operation
krb5kdc: starting...
W20260619 14:34:38.966241 2056 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Jun 19 14:34:39 dist-test-slave-70rm krb5kdc[15366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:34:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:34:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.2.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab.
Entry for principal kudu/127.2.2.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab.
I20260619 14:34:39.089354 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:41727
--webserver_interface=127.2.2.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:38521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.2.62:41727,127.2.2.61:45191,127.2.2.60:33569
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:34:39.201045 15379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:34:39.201395 15379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:34:39.201473 15379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:34:39.205787 15379 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:34:39.205883 15379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:34:39.205921 15379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:34:39.205981 15379 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:34:39.206015 15379 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:34:39.211501 15379 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:38521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.2.62:41727,127.2.2.61:45191,127.2.2.60:33569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:41727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.2.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:34:39.212657 15379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:34:39.213694 15379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:34:39.222360 15385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:34:39.222360 15387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:34:39.222445 15384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:34:39.222568 15379 server_base.cc:1034] running on GCE node
I20260619 14:34:39.223312 15379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:34:39.224498 15379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:34:39.225688 15379 hybrid_clock.cc:648] HybridClock initialized: now 1781879679225661 us; error 44 us; skew 500 ppm
Jun 19 14:34:39 dist-test-slave-70rm krb5kdc[15366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879679, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:34:39.228679 15379 init.cc:377] Logged in from keytab as kudu/127.2.2.62@KRBTEST.COM (short username kudu)
I20260619 14:34:39.229684 15379 webserver.cc:460] Webserver started at http://127.2.2.62:38241/ using document root <none> and password file <none>
I20260619 14:34:39.230316 15379 fs_manager.cc:359] Metadata directory not provided
I20260619 14:34:39.230384 15379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:34:39.230554 15379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:34:39.232537 15379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/minicluster-data/master-0/data/instance:
uuid: "6d33269977be4859a11e98dacc85ad23"
format_stamp: "Formatted at 2026-06-19 14:34:39 on dist-test-slave-70rm"
server_key: "2c41873baa1d2dff1f706b0847273db3"
server_key_iv: "95b6dbdc3b58d430f6770811fe29952d"
server_key_version: "encryptionkey@0"
I20260619 14:34:39.233054 15379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/minicluster-data/master-0/wal/instance:
uuid: "6d33269977be4859a11e98dacc85ad23"
format_stamp: "Formatted at 2026-06-19 14:34:39 on dist-test-slave-70rm"
server_key: "2c41873baa1d2dff1f706b0847273db3"
server_key_iv: "95b6dbdc3b58d430f6770811fe29952d"
server_key_version: "encryptionkey@0"
I20260619 14:34:39.236677 15379 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:34:39.239025 15394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:34:39.240229 15379 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:34:39.240389 15379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/minicluster-data/master-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/minicluster-data/master-0/wal
uuid: "6d33269977be4859a11e98dacc85ad23"
format_stamp: "Formatted at 2026-06-19 14:34:39 on dist-test-slave-70rm"
server_key: "2c41873baa1d2dff1f706b0847273db3"
server_key_iv: "95b6dbdc3b58d430f6770811fe29952d"
server_key_version: "encryptionkey@0"
I20260619 14:34:39.240495 15379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:34:39.255944 15379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:34:39.258862 15379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:34:39.259073 15379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:34:39.266028 15379 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.62:41727
I20260619 14:34:39.266053 15446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.62:41727 every 8 connection(s)
I20260619 14:34:39.267102 15379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
I20260619 14:34:39.270045 15447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:34:39.273959 15447 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41727 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:34:39.276094 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 15379
I20260619 14:34:39.276235 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/minicluster-data/master-0/wal/instance
I20260619 14:34:39.276540 2056 external_mini_cluster.cc:1428] Setting key 066bad11803707d5355a41226d0d1799
Jun 19 14:34:39 dist-test-slave-70rm krb5kdc[15366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879679, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.62@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.2.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.61@KRBTEST.COM" created.
I20260619 14:34:39.288862 15450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:39.276264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52214 (local address 127.2.2.62:41727)
0619 14:34:39.276929 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0619 14:34:39.276943 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:34:39.276993 (+ 50us) server_negotiation.cc:407] Connection header received
0619 14:34:39.277981 (+ 988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:34:39.277986 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:34:39.278254 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:34:39.278458 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:34:39.279307 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:39.280180 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:39.281154 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:39.281533 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:39.281775 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:39.284181 (+ 2406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:34:39.284187 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:34:39.284200 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:34:39.284229 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:34:39.285713 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:39.286238 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:39.286244 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:39.286245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:39.286411 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:39.286738 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:39.286741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:39.286743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:39.287140 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:34:39.287277 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 14:34:39.287732 (+ 455us) server_negotiation.cc:299] Negotiation successful
0619 14:34:39.287999 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":527,"spinlock_wait_cycles":13568,"thread_start_us":159,"threads_started":1}
I20260619 14:34:39.288869 15449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:39.276264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41727 (local address 127.0.0.1:52214)
0619 14:34:39.276785 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:39.276838 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 14:34:39.277677 (+ 839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:34:39.278451 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:34:39.278456 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:34:39.278916 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:34:39.279160 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:39.279173 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:39.280308 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:39.280313 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:39.281032 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:39.281040 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:39.281640 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:39.281693 (+ 53us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:39.281853 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:39.282531 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:34:39.282548 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:34:39.284031 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:34:39.285832 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:34:39.285843 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:34:39.285853 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:34:39.286083 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:34:39.286506 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:34:39.286511 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:34:39.286513 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:34:39.286635 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:34:39.287259 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:34:39.287265 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:34:39.287404 (+ 139us) client_negotiation.cc:769] Sending connection context
0619 14:34:39.287596 (+ 192us) client_negotiation.cc:240] Negotiation successful
0619 14:34:39.288100 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":157,"thread_start_us":135,"threads_started":1}
I20260619 14:34:39.291409 15447 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.2.61" port: 45191 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:34:39.292199 15449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:39.291934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.61:45191 (local address 127.0.0.1:56632)
0619 14:34:39.292063 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:39.292134 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.61:45191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260619 14:34:39.292474 15397 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.2.61:45191: connect: Connection refused (error 111)
I20260619 14:34:39.293313 15449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:39.293122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.61:45191 (local address 127.0.0.1:56644)
0619 14:34:39.293219 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:39.293263 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.61:45191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 14:34:39.293537 15447 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.2.61:45191: Network error: Client connection negotiation failed: client connection to 127.2.2.61:45191: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.61.keytab.
Entry for principal kudu/127.2.2.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.61.keytab.
I20260619 14:34:39.302778 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.61:45191
--webserver_interface=127.2.2.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.61.keytab
--principal=kudu/127.2.2.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:38521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.2.62:41727,127.2.2.61:45191,127.2.2.60:33569
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:34:39.342792 15447 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.2.61" port: 45191 } attempt: 1
I20260619 14:34:39.343737 15449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:39.343398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.61:45191 (local address 127.0.0.1:56656)
0619 14:34:39.343587 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:39.343658 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.61:45191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260619 14:34:39.344589 15449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:39.344356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.61:45191 (local address 127.0.0.1:56660)
0619 14:34:39.344465 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:39.344520 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.61:45191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 14:34:39.344738 15447 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.2.61:45191: Network error: Client connection negotiation failed: client connection to 127.2.2.61:45191: connect: Connection refused (error 111)
W20260619 14:34:39.412531 15454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:34:39.412858 15454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:34:39.412931 15454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 14:34:39.412962 15447 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.2.61" port: 45191 } attempt: 2
I20260619 14:34:39.413848 15449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:39.413543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.61:45191 (local address 127.0.0.1:56670)
0619 14:34:39.413699 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:39.413764 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.61:45191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260619 14:34:39.414558 15449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:39.414321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.61:45191 (local address 127.0.0.1:56672)
0619 14:34:39.414449 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:39.414499 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.61:45191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260619 14:34:39.414683 15447 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.2.61:45191: Network error: Client connection negotiation failed: client connection to 127.2.2.61:45191: connect: Connection refused (error 111)
W20260619 14:34:39.417353 15454 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:34:39.417408 15454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:34:39.417428 15454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:34:39.417444 15454 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:34:39.417459 15454 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:34:39.422845 15454 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:38521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.2.62:41727,127.2.2.61:45191,127.2.2.60:33569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.61.keytab
--principal=kudu/127.2.2.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.61:45191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.2.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:34:39.423895 15454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:34:39.424928 15454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:34:39.433501 15462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:34:39.433501 15460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:34:39.433501 15459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:34:39.433768 15454 server_base.cc:1034] running on GCE node
I20260619 14:34:39.434492 15454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:34:39.435642 15454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:34:39.436797 15454 hybrid_clock.cc:648] HybridClock initialized: now 1781879679436781 us; error 31 us; skew 500 ppm
Jun 19 14:34:39 dist-test-slave-70rm krb5kdc[15366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879679, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:34:39.439561 15454 init.cc:377] Logged in from keytab as kudu/127.2.2.61@KRBTEST.COM (short username kudu)
I20260619 14:34:39.440488 15454 webserver.cc:460] Webserver started at http://127.2.2.61:36141/ using document root <none> and password file <none>
I20260619 14:34:39.441048 15454 fs_manager.cc:359] Metadata directory not provided
I20260619 14:34:39.441151 15454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:34:39.441385 15454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:34:39.443886 15454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/minicluster-data/master-1/data/instance:
uuid: "8202526897ed4dd9ac5c03333b95bc18"
format_stamp: "Formatted at 2026-06-19 14:34:39 on dist-test-slave-70rm"
server_key: "3c2fc2d210b596a587a6f238438af074"
server_key_iv: "51404dc8dab327aa21030b34aac2688c"
server_key_version: "encryptionkey@0"
I20260619 14:34:39.444505 15454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/minicluster-data/master-1/wal/instance:
uuid: "8202526897ed4dd9ac5c03333b95bc18"
format_stamp: "Formatted at 2026-06-19 14:34:39 on dist-test-slave-70rm"
server_key: "3c2fc2d210b596a587a6f238438af074"
server_key_iv: "51404dc8dab327aa21030b34aac2688c"
server_key_version: "encryptionkey@0"
I20260619 14:34:39.448249 15454 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:34:39.450709 15469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:34:39.451898 15454 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:34:39.452042 15454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/minicluster-data/master-1/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/minicluster-data/master-1/wal
uuid: "8202526897ed4dd9ac5c03333b95bc18"
format_stamp: "Formatted at 2026-06-19 14:34:39 on dist-test-slave-70rm"
server_key: "3c2fc2d210b596a587a6f238438af074"
server_key_iv: "51404dc8dab327aa21030b34aac2688c"
server_key_version: "encryptionkey@0"
I20260619 14:34:39.452167 15454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:34:39.473016 15454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:34:39.476048 15454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:34:39.476287 15454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:34:39.483655 15454 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.61:45191
I20260619 14:34:39.483682 15521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.61:45191 every 8 connection(s)
I20260619 14:34:39.484625 15454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/minicluster-data/master-1/data/info.pb
I20260619 14:34:39.487385 15522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:34:39.489195 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 15454
I20260619 14:34:39.489322 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/minicluster-data/master-1/wal/instance
I20260619 14:34:39.489612 2056 external_mini_cluster.cc:1428] Setting key 1605e8f83a9fbc8fad8cd81269a0da5e
I20260619 14:34:39.492309 15522 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41727 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.2.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.60@KRBTEST.COM" created.
Jun 19 14:34:39 dist-test-slave-70rm krb5kdc[15366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879679, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.61@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:34:39.505841 15447 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.2.61" port: 45191 } attempt: 3
I20260619 14:34:39.508097 15450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:39.494388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52224 (local address 127.2.2.62:41727)
0619 14:34:39.494550 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 14:34:39.494554 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:34:39.495636 (+ 1082us) server_negotiation.cc:407] Connection header received
0619 14:34:39.496634 (+ 998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:34:39.496637 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:34:39.496706 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:34:39.496804 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:34:39.498023 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:39.498667 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:39.499964 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:39.500270 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:39.500392 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:39.503237 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:34:39.503241 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:34:39.503244 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:34:39.503280 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:34:39.504899 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:39.505469 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:39.505473 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:39.505475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:39.505564 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:39.505993 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:39.505999 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:39.506001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:39.506297 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:34:39.506414 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 14:34:39.507218 (+ 804us) server_negotiation.cc:299] Negotiation successful
0619 14:34:39.507519 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 14:34:39.508831 15525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:39.494948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41727 (local address 127.0.0.1:52224)
0619 14:34:39.495478 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:39.495522 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 14:34:39.496411 (+ 889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:34:39.496984 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:34:39.496991 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:34:39.497332 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:34:39.497872 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:39.497885 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:39.498805 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:39.498809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:39.499831 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:39.499840 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:39.500383 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:39.500436 (+ 53us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:39.500611 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:39.501232 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:34:39.501248 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:34:39.503058 (+ 1810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:34:39.505034 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:34:39.505041 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:34:39.505059 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:34:39.505346 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:34:39.505660 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:34:39.505665 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:34:39.505667 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:34:39.505856 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:34:39.506478 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:34:39.506486 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:34:39.506880 (+ 394us) client_negotiation.cc:769] Sending connection context
0619 14:34:39.507347 (+ 467us) client_negotiation.cc:240] Negotiation successful
0619 14:34:39.507756 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":149,"threads_started":1}
I20260619 14:34:39.510663 15522 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.2.61" port: 45191 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:34:39 dist-test-slave-70rm krb5kdc[15366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879679, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.62@KRBTEST.COM for kudu/127.2.2.61@KRBTEST.COM
Jun 19 14:34:39 dist-test-slave-70rm krb5kdc[15366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879679, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.61@KRBTEST.COM for kudu/127.2.2.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.60.keytab.
Entry for principal kudu/127.2.2.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.60.keytab.
I20260619 14:34:39.516921 15449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:39.506354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.61:45191 (local address 127.0.0.1:56674)
0619 14:34:39.506566 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:39.506581 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:34:39.506664 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:34:39.507272 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:34:39.507275 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:34:39.507507 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:34:39.507629 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:39.507636 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:39.508954 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:39.508958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:39.509633 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:39.509642 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:39.510564 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:39.510580 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:39.510704 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:39.511291 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:34:39.511305 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:34:39.512865 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:34:39.515243 (+ 2378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:34:39.515247 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:34:39.515249 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:34:39.515504 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:34:39.515807 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:34:39.515810 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:34:39.515812 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:34:39.515902 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:34:39.516453 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:34:39.516457 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:34:39.516578 (+ 121us) client_negotiation.cc:769] Sending connection context
0619 14:34:39.516681 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 14:34:39.516795 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132}
I20260619 14:34:39.517233 15527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:39.506482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56674 (local address 127.2.2.61:45191)
0619 14:34:39.506900 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0619 14:34:39.506913 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:34:39.506931 (+ 18us) server_negotiation.cc:407] Connection header received
0619 14:34:39.506972 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:34:39.506976 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:34:39.507144 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:34:39.507293 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:34:39.507767 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:39.508826 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:39.509892 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:39.510456 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:39.510581 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:39.513015 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:34:39.513025 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:34:39.513041 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:34:39.513064 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:34:39.515131 (+ 2067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:39.515619 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:39.515626 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:39.515627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:39.515711 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:39.515991 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:39.515997 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:39.515998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:39.516328 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:34:39.516467 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 14:34:39.516799 (+ 332us) server_negotiation.cc:299] Negotiation successful
0619 14:34:39.516978 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":163,"threads_started":1}
I20260619 14:34:39.517833 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.60:33569
--webserver_interface=127.2.2.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.60.keytab
--principal=kudu/127.2.2.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:38521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.2.62:41727,127.2.2.61:45191,127.2.2.60:33569
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:34:39.520368 15525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:39.511247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.61:45191 (local address 127.0.0.1:56676)
0619 14:34:39.511377 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:39.511392 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:34:39.511476 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:34:39.511938 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:34:39.511960 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:34:39.512137 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:34:39.512266 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:39.512290 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:39.513245 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:39.513247 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:39.513956 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:39.513970 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:39.514595 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:39.514609 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:39.514706 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:39.515160 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:34:39.515171 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:34:39.516478 (+ 1307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:34:39.518748 (+ 2270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:34:39.518754 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:34:39.518759 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:34:39.519044 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:34:39.519366 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:34:39.519371 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:34:39.519372 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:34:39.519477 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:34:39.519843 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:34:39.519846 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:34:39.519925 (+ 79us) client_negotiation.cc:769] Sending connection context
0619 14:34:39.520039 (+ 114us) client_negotiation.cc:240] Negotiation successful
0619 14:34:39.520196 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":3}
I20260619 14:34:39.520375 15528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:39.511343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56676 (local address 127.2.2.61:45191)
0619 14:34:39.511690 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0619 14:34:39.511702 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:34:39.511717 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:34:39.511757 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:34:39.511761 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:34:39.511815 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:34:39.511904 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:34:39.512460 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:39.513133 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:39.514107 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:39.514491 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:39.514601 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:39.516596 (+ 1995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:34:39.516599 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:34:39.516600 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:34:39.516618 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:34:39.518639 (+ 2021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:39.519158 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:39.519163 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:39.519165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:39.519261 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:39.519570 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:39.519576 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:39.519578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:39.519749 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:34:39.519854 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 14:34:39.520099 (+ 245us) server_negotiation.cc:299] Negotiation successful
0619 14:34:39.520249 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":78,"server-negotiator.queue_time_us":279,"thread_start_us":144,"threads_started":1}
I20260619 14:34:39.521440 15522 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.2.60" port: 33569 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:34:39.521453 15447 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.2.60" port: 33569 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:34:39.522214 15525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:39.521899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.60:33569 (local address 127.0.0.1:38504)
0619 14:34:39.522030 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:39.522110 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.60:33569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 14:34:39.522254 15449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:39.521930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.60:33569 (local address 127.0.0.1:38508)
0619 14:34:39.522142 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:39.522189 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.60:33569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
W20260619 14:34:39.522428 15472 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.2.60:33569: connect: Connection refused (error 111)
I20260619 14:34:39.523022 15449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:39.522707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.60:33569 (local address 127.0.0.1:38522)
0619 14:34:39.522898 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:39.522956 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.60:33569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
W20260619 14:34:39.523187 15447 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.2.60:33569: Network error: Client connection negotiation failed: client connection to 127.2.2.60:33569: connect: Connection refused (error 111)
I20260619 14:34:39.523411 15525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:39.523119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.60:33569 (local address 127.0.0.1:38532)
0619 14:34:39.523276 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:39.523331 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.60:33569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260619 14:34:39.523564 15522 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.2.60:33569: Network error: Client connection negotiation failed: client connection to 127.2.2.60:33569: connect: Connection refused (error 111)
I20260619 14:34:39.554381 15447 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.2.60" port: 33569 } attempt: 1
I20260619 14:34:39.555347 15449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:39.555011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.60:33569 (local address 127.0.0.1:38544)
0619 14:34:39.555165 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:39.555239 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.60:33569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 14:34:39.556217 15449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:39.555995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.60:33569 (local address 127.0.0.1:38546)
0619 14:34:39.556106 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:39.556154 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.60:33569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 14:34:39.556421 15447 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.2.60:33569: Network error: Client connection negotiation failed: client connection to 127.2.2.60:33569: connect: Connection refused (error 111)
I20260619 14:34:39.572875 15522 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.2.60" port: 33569 } attempt: 1
I20260619 14:34:39.573812 15525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:39.573458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.60:33569 (local address 127.0.0.1:38562)
0619 14:34:39.573644 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:39.573719 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.60:33569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260619 14:34:39.574649 15525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:39.574412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.60:33569 (local address 127.0.0.1:38566)
0619 14:34:39.574523 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:39.574575 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.60:33569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 14:34:39.574815 15522 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.2.60:33569: Network error: Client connection negotiation failed: client connection to 127.2.2.60:33569: connect: Connection refused (error 111)
I20260619 14:34:39.631578 15447 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.2.60" port: 33569 } attempt: 2
I20260619 14:34:39.632419 15449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:39.632129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.60:33569 (local address 127.0.0.1:38580)
0619 14:34:39.632275 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:39.632342 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.60:33569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 14:34:39.633145 15449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:39.632924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.60:33569 (local address 127.0.0.1:38592)
0619 14:34:39.633037 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:39.633092 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.60:33569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 14:34:39.633278 15447 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.2.60:33569: Network error: Client connection negotiation failed: client connection to 127.2.2.60:33569: connect: Connection refused (error 111)
W20260619 14:34:39.635567 15529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:34:39.635859 15529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:34:39.635960 15529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:34:39.640125 15529 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:34:39.640209 15529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:34:39.640259 15529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:34:39.640296 15529 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:34:39.640328 15529 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:34:39.643025 15522 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.2.60" port: 33569 } attempt: 2
I20260619 14:34:39.643973 15525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:39.643602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.60:33569 (local address 127.0.0.1:38600)
0619 14:34:39.643802 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:39.643889 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.60:33569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260619 14:34:39.644759 15525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:39.644535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.60:33569 (local address 127.0.0.1:38604)
0619 14:34:39.644649 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:39.644699 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.60:33569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 14:34:39.644889 15522 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.2.60:33569: Network error: Client connection negotiation failed: client connection to 127.2.2.60:33569: connect: Connection refused (error 111)
I20260619 14:34:39.646433 15529 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:38521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.2.62:41727,127.2.2.61:45191,127.2.2.60:33569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.60.keytab
--principal=kudu/127.2.2.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.60:33569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.2.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:34:39.647527 15529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:34:39.648558 15529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:34:39.657050 15538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:34:39.657050 15536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:34:39.657308 15535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:34:39.657577 15529 server_base.cc:1034] running on GCE node
I20260619 14:34:39.658267 15529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:34:39.659432 15529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:34:39.660629 15529 hybrid_clock.cc:648] HybridClock initialized: now 1781879679660614 us; error 33 us; skew 500 ppm
Jun 19 14:34:39 dist-test-slave-70rm krb5kdc[15366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879679, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:34:39.663462 15529 init.cc:377] Logged in from keytab as kudu/127.2.2.60@KRBTEST.COM (short username kudu)
I20260619 14:34:39.664431 15529 webserver.cc:460] Webserver started at http://127.2.2.60:45949/ using document root <none> and password file <none>
I20260619 14:34:39.665004 15529 fs_manager.cc:359] Metadata directory not provided
I20260619 14:34:39.665104 15529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:34:39.665338 15529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:34:39.667482 15529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/minicluster-data/master-2/data/instance:
uuid: "46841f5ad9af4fb99fc13a8d3318db4c"
format_stamp: "Formatted at 2026-06-19 14:34:39 on dist-test-slave-70rm"
server_key: "0847bae5130debb8b1f96400ecfe4601"
server_key_iv: "fcc48db66359b529f0967092331e08e5"
server_key_version: "encryptionkey@0"
I20260619 14:34:39.668063 15529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/minicluster-data/master-2/wal/instance:
uuid: "46841f5ad9af4fb99fc13a8d3318db4c"
format_stamp: "Formatted at 2026-06-19 14:34:39 on dist-test-slave-70rm"
server_key: "0847bae5130debb8b1f96400ecfe4601"
server_key_iv: "fcc48db66359b529f0967092331e08e5"
server_key_version: "encryptionkey@0"
I20260619 14:34:39.671880 15529 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:34:39.674369 15545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:34:39.675588 15529 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:34:39.675729 15529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/minicluster-data/master-2/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/minicluster-data/master-2/wal
uuid: "46841f5ad9af4fb99fc13a8d3318db4c"
format_stamp: "Formatted at 2026-06-19 14:34:39 on dist-test-slave-70rm"
server_key: "0847bae5130debb8b1f96400ecfe4601"
server_key_iv: "fcc48db66359b529f0967092331e08e5"
server_key_version: "encryptionkey@0"
I20260619 14:34:39.675845 15529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:34:39.708782 15529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:34:39.711834 15529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:34:39.712044 15529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:34:39.719198 15529 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.60:33569
I20260619 14:34:39.719218 15597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.60:33569 every 8 connection(s)
I20260619 14:34:39.720180 15529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/minicluster-data/master-2/data/info.pb
I20260619 14:34:39.723047 15598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:34:39.725463 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 15529
I20260619 14:34:39.725606 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879514993456-2056-0/minicluster-data/master-2/wal/instance
I20260619 14:34:39.725924 2056 external_mini_cluster.cc:1428] Setting key 226d90cf3927c1929bd34e2ac6d46c2b
I20260619 14:34:39.727489 15598 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41727 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:34:39.732416 15447 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.2.60" port: 33569 } attempt: 3
Jun 19 14:34:39 dist-test-slave-70rm krb5kdc[15366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879679, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.60@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:34:39.736040 15522 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.2.60" port: 33569 } attempt: 3
Jun 19 14:34:39 dist-test-slave-70rm krb5kdc[15366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.2.60@KRBTEST.COM
Jun 19 14:34:39 dist-test-slave-70rm krb5kdc[15366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879679, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.62@KRBTEST.COM for kudu/127.2.2.60@KRBTEST.COM
I20260619 14:34:39.742663 15601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:39.727315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38614 (local address 127.2.2.60:33569)
0619 14:34:39.727726 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0619 14:34:39.727741 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:34:39.727776 (+ 35us) server_negotiation.cc:407] Connection header received
0619 14:34:39.728462 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:34:39.728480 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:34:39.728829 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:34:39.729215 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:34:39.730116 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:39.731277 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:39.732404 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:39.733435 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:39.733758 (+ 323us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:39.737206 (+ 3448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:34:39.737212 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:34:39.737227 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:34:39.737252 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:34:39.739297 (+ 2045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:39.739883 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:39.739889 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:39.739891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:39.740045 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:39.740372 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:39.740375 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:39.740378 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:39.740881 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:34:39.741079 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0619 14:34:39.741901 (+ 822us) server_negotiation.cc:299] Negotiation successful
0619 14:34:39.742047 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":75,"server-negotiator.queue_time_us":269,"spinlock_wait_cycles":2816,"thread_start_us":139,"threads_started":1}
I20260619 14:34:39.743799 15603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:39.732953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38620 (local address 127.2.2.60:33569)
0619 14:34:39.733326 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0619 14:34:39.733342 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 14:34:39.733356 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:34:39.733386 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:34:39.733388 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:34:39.733428 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:34:39.733524 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:34:39.734169 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:39.734801 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:39.735574 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:39.735829 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:39.735914 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:39.739066 (+ 3152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:34:39.739069 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:34:39.739071 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:34:39.739093 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:34:39.741430 (+ 2337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:39.741909 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:39.741914 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:39.741919 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:39.742012 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:39.742324 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:39.742328 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:39.742330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:39.742553 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:34:39.743308 (+ 755us) server_negotiation.cc:1035] Waiting for connection context
0619 14:34:39.743390 (+ 82us) server_negotiation.cc:299] Negotiation successful
0619 14:34:39.743556 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":298,"thread_start_us":101,"threads_started":1}
I20260619 14:34:39.744207 15449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:39.732898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.60:33569 (local address 127.0.0.1:38620)
0619 14:34:39.733028 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:39.733041 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:34:39.733105 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:34:39.733545 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:34:39.733548 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:34:39.733745 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:34:39.733837 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:39.733844 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:39.734925 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:39.734929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:39.735467 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:39.735477 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:39.735929 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:39.735957 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:39.736057 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:39.736557 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:34:39.736575 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:34:39.738914 (+ 2339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:34:39.741562 (+ 2648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:34:39.741566 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:34:39.741568 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:34:39.741796 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:34:39.742108 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:34:39.742111 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:34:39.742113 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:34:39.742221 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:34:39.742656 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:34:39.742659 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:34:39.742787 (+ 128us) client_negotiation.cc:769] Sending connection context
0619 14:34:39.742880 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 14:34:39.743036 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"spinlock_wait_cycles":768}
I20260619 14:34:39.742663 15602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:39.729276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41727 (local address 127.0.0.1:52238)
0619 14:34:39.729823 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:39.729882 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 14:34:39.730013 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:34:39.730372 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:34:39.730377 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:34:39.730725 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:34:39.730897 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:39.730907 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:39.731755 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:39.731758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:39.732832 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:39.732843 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:39.733385 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:39.733437 (+ 52us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:39.733760 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:39.734525 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:34:39.734551 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:34:39.736823 (+ 2272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:34:39.739153 (+ 2330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:34:39.739163 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:34:39.739178 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:34:39.739509 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:34:39.739861 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:34:39.739864 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:34:39.739866 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:34:39.740045 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:34:39.740634 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:34:39.740641 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:34:39.740880 (+ 239us) client_negotiation.cc:769] Sending connection context
0619 14:34:39.741080 (+ 200us) client_negotiation.cc:240] Negotiation successful
0619 14:34:39.741475 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"mutex_wait_us":136,"spinlock_wait_cycles":52096,"thread_start_us":134,"threads_started":1}
Jun 19 14:34:39 dist-test-slave-70rm krb5kdc[15366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879679, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.61@KRBTEST.COM for kudu/127.2.2.60@KRBTEST.COM
I20260619 14:34:39.744213 15450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:39.729361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52238 (local address 127.2.2.62:41727)
0619 14:34:39.729546 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 14:34:39.729551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:34:39.729980 (+ 429us) server_negotiation.cc:407] Connection header received
0619 14:34:39.730154 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:34:39.730158 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:34:39.730210 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:34:39.730301 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:34:39.731047 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:39.731635 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:39.732954 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:39.733276 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:39.733379 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:39.736993 (+ 3614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:34:39.736997 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:34:39.737000 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:34:39.737030 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:34:39.739009 (+ 1979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:39.739644 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:39.739649 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:39.739651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:39.739742 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:39.740167 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:39.740171 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:39.740172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:39.740485 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:34:39.740583 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 14:34:39.742774 (+ 2191us) server_negotiation.cc:299] Negotiation successful
0619 14:34:39.742933 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260619 14:34:39.746568 15598 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.2.61" port: 45191 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:34:39.748494 15447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d33269977be4859a11e98dacc85ad23: Bootstrap starting.
I20260619 14:34:39.749763 15525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:39.736528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.60:33569 (local address 127.0.0.1:38624)
0619 14:34:39.737070 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:39.737089 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 14:34:39.737173 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:34:39.738805 (+ 1632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:34:39.738810 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:34:39.739029 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:34:39.739152 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:39.739160 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:39.741008 (+ 1848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:39.741014 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:39.741876 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:39.741889 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:39.742508 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:39.742527 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:39.742631 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:39.743156 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:34:39.743170 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:34:39.744833 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:34:39.747968 (+ 3135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:34:39.747972 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:34:39.747975 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:34:39.748263 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:34:39.748600 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:34:39.748603 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:34:39.748605 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:34:39.748714 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:34:39.749167 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:34:39.749173 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:34:39.749267 (+ 94us) client_negotiation.cc:769] Sending connection context
0619 14:34:39.749385 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 14:34:39.749543 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442}
I20260619 14:34:39.750349 15604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:39.737203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38624 (local address 127.2.2.60:33569)
0619 14:34:39.738376 (+ 1173us) server_negotiation.cc:206] Beginning negotiation
0619 14:34:39.738390 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:34:39.738406 (+ 16us) server_negotiation.cc:407] Connection header received
0619 14:34:39.738438 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:34:39.738441 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:34:39.738634 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:34:39.739274 (+ 640us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:34:39.739687 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:39.740848 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:39.742022 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:39.742394 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:39.743803 (+ 1409us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:39.745705 (+ 1902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:34:39.745711 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:34:39.745714 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:34:39.745738 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:34:39.747834 (+ 2096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:39.748382 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:39.748386 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:39.748388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:39.748475 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:39.748826 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:39.748832 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:39.748835 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:39.749040 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:34:39.749158 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 14:34:39.749979 (+ 821us) server_negotiation.cc:299] Negotiation successful
0619 14:34:39.750107 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1102,"thread_start_us":554,"threads_started":1}
Jun 19 14:34:39 dist-test-slave-70rm krb5kdc[15366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:34:39.753670 15522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8202526897ed4dd9ac5c03333b95bc18: Bootstrap starting.
I20260619 14:34:39.755574 15447 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d33269977be4859a11e98dacc85ad23: Neither blocks nor log segments found. Creating new log.
I20260619 14:34:39.756011 15450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:39.745616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52254 (local address 127.2.2.62:41727)
0619 14:34:39.745963 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0619 14:34:39.745970 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:34:39.745985 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:34:39.746028 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:34:39.746032 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:34:39.746075 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:34:39.746190 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:34:39.747125 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:39.747927 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:39.748889 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:39.749244 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:39.749358 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:39.751934 (+ 2576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:34:39.751939 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:34:39.751942 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:34:39.751973 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:34:39.753605 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:39.754051 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:39.754055 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:39.754057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:39.754186 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:39.754507 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:39.754511 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:39.754514 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:39.754745 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:34:39.754914 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 14:34:39.755372 (+ 458us) server_negotiation.cc:299] Negotiation successful
0619 14:34:39.755507 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272}
I20260619 14:34:39.756623 15522 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8202526897ed4dd9ac5c03333b95bc18: Neither blocks nor log segments found. Creating new log.
I20260619 14:34:39.757033 15447 log.cc:824] T 00000000000000000000000000000000 P 6d33269977be4859a11e98dacc85ad23: Log is configured to *not* fsync() on all Append() calls
I20260619 14:34:39.757637 15522 log.cc:824] T 00000000000000000000000000000000 P 8202526897ed4dd9ac5c03333b95bc18: Log is configured to *not* fsync() on all Append() calls
I20260619 14:34:39.759907 15522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8202526897ed4dd9ac5c03333b95bc18: No bootstrap required, opened a new log
Jun 19 14:34:39 dist-test-slave-70rm krb5kdc[15366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879679, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.60@KRBTEST.COM for kudu/127.2.2.61@KRBTEST.COM
I20260619 14:34:39.760020 15447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d33269977be4859a11e98dacc85ad23: No bootstrap required, opened a new log
Jun 19 14:34:39 dist-test-slave-70rm krb5kdc[15366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.2.61@KRBTEST.COM
I20260619 14:34:39.764475 15528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:39.754319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56682 (local address 127.2.2.61:45191)
0619 14:34:39.754465 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 14:34:39.754470 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:34:39.754485 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:34:39.754524 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:34:39.754527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:34:39.754574 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:34:39.754654 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:34:39.755337 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:39.756166 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:39.757508 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:39.757882 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:39.757998 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:39.760562 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:34:39.760566 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:34:39.760569 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:34:39.760597 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:34:39.762444 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:39.763003 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:39.763007 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:39.763009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:39.763099 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:39.763429 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:39.763432 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:39.763434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:39.763694 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:34:39.763775 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 14:34:39.764178 (+ 403us) server_negotiation.cc:299] Negotiation successful
0619 14:34:39.764311 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 14:34:39.764680 15522 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8202526897ed4dd9ac5c03333b95bc18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d33269977be4859a11e98dacc85ad23" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41727 } } peers { permanent_uuid: "8202526897ed4dd9ac5c03333b95bc18" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 45191 } } peers { permanent_uuid: "46841f5ad9af4fb99fc13a8d3318db4c" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 33569 } }
I20260619 14:34:39.764932 15522 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8202526897ed4dd9ac5c03333b95bc18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:34:39.764991 15522 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8202526897ed4dd9ac5c03333b95bc18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8202526897ed4dd9ac5c03333b95bc18, State: Initialized, Role: FOLLOWER
I20260619 14:34:39.765012 15602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:39.747064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.61:45191 (local address 127.0.0.1:56682)
0619 14:34:39.747172 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:39.747187 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:34:39.747264 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:34:39.754778 (+ 7514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:34:39.754782 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:34:39.755020 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:34:39.755193 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:39.755204 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:39.756371 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:39.756374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:39.757343 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:39.757357 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:39.758187 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:39.758204 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:39.758316 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:39.758805 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:34:39.758820 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:34:39.760394 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:34:39.762583 (+ 2189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:34:39.762587 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:34:39.762589 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:34:39.762862 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:34:39.763217 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:34:39.763221 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:34:39.763223 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:34:39.763318 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:34:39.763855 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:34:39.763858 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:34:39.763974 (+ 116us) client_negotiation.cc:769] Sending connection context
0619 14:34:39.764668 (+ 694us) client_negotiation.cc:240] Negotiation successful
0619 14:34:39.764827 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260619 14:34:39.764879 15447 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d33269977be4859a11e98dacc85ad23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d33269977be4859a11e98dacc85ad23" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41727 } } peers { permanent_uuid: "8202526897ed4dd9ac5c03333b95bc18" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 45191 } } peers { permanent_uuid: "46841f5ad9af4fb99fc13a8d3318db4c" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 33569 } }
I20260619 14:34:39.765115 15447 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d33269977be4859a11e98dacc85ad23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:34:39.765182 15447 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d33269977be4859a11e98dacc85ad23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d33269977be4859a11e98dacc85ad23, State: Initialized, Role: FOLLOWER
I20260619 14:34:39.765758 15598 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.2.60" port: 33569 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:34:39.765753 15447 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d33269977be4859a11e98dacc85ad23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d33269977be4859a11e98dacc85ad23" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41727 } } peers { permanent_uuid: "8202526897ed4dd9ac5c03333b95bc18" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 45191 } } peers { permanent_uuid: "46841f5ad9af4fb99fc13a8d3318db4c" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 33569 } }
I20260619 14:34:39.766285 15522 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8202526897ed4dd9ac5c03333b95bc18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d33269977be4859a11e98dacc85ad23" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41727 } } peers { permanent_uuid: "8202526897ed4dd9ac5c03333b95bc18" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 45191 } } peers { permanent_uuid: "46841f5ad9af4fb99fc13a8d3318db4c" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 33569 } }
I20260619 14:34:39.767442 15447 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d33269977be4859a11e98dacc85ad23 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:34:39.767935 15522 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8202526897ed4dd9ac5c03333b95bc18 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:34:39.768491 15527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:39.757355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56696 (local address 127.2.2.61:45191)
0619 14:34:39.757495 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 14:34:39.757500 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:34:39.757655 (+ 155us) server_negotiation.cc:407] Connection header received
0619 14:34:39.757798 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:34:39.757801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:34:39.757849 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:34:39.757889 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:34:39.759070 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:39.759752 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:39.761172 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:39.761766 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:39.761842 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:39.764763 (+ 2921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:34:39.764766 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:34:39.764769 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:34:39.764793 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:34:39.766513 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:39.767054 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:39.767057 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:39.767059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:39.767143 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:39.767439 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:39.767443 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:39.767454 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:39.767663 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:34:39.767826 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 14:34:39.768225 (+ 399us) server_negotiation.cc:299] Negotiation successful
0619 14:34:39.768353 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Jun 19 14:34:39 dist-test-slave-70rm krb5kdc[15366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879679, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.60@KRBTEST.COM for kudu/127.2.2.60@KRBTEST.COM
I20260619 14:34:39.772554 2056 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:34:39.776367 15604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:39.766619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38630 (local address 127.2.2.60:33569)
0619 14:34:39.766746 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 14:34:39.766751 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:34:39.766810 (+ 59us) server_negotiation.cc:407] Connection header received
0619 14:34:39.766979 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:34:39.766983 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:34:39.767023 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:34:39.767072 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:34:39.767607 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:39.768468 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:39.769691 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:39.770165 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:39.770250 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:39.772696 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:34:39.772700 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:34:39.772703 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:34:39.772731 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:34:39.774495 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:39.775018 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:39.775024 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:39.775026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:39.775113 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:39.775452 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:39.775455 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:39.775457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:39.775674 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:34:39.775772 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 14:34:39.776078 (+ 306us) server_negotiation.cc:299] Negotiation successful
0619 14:34:39.776204 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 14:34:39.776795 15602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:39.766285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.60:33569 (local address 127.0.0.1:38630)
0619 14:34:39.766742 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:39.766758 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:34:39.766868 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:34:39.767193 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:34:39.767196 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:34:39.767382 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:34:39.767482 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:39.767489 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:39.768613 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:39.768616 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:39.769552 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:39.769563 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:39.770379 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:39.770394 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:39.770487 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:39.770949 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:34:39.770964 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:34:39.772552 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:34:39.774620 (+ 2068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:34:39.774624 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:34:39.774630 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:34:39.774907 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:34:39.775223 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:34:39.775226 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:34:39.775232 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:34:39.775329 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:34:39.775810 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:34:39.775815 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:34:39.775901 (+ 86us) client_negotiation.cc:769] Sending connection context
0619 14:34:39.776504 (+ 603us) client_negotiation.cc:240] Negotiation successful
0619 14:34:39.776641 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":97,"threads_started":1}
W20260619 14:34:39.778676 15631 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8202526897ed4dd9ac5c03333b95bc18: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:34:39.778869 15631 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:34:39.779134 15609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8202526897ed4dd9ac5c03333b95bc18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d33269977be4859a11e98dacc85ad23" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41727 } } peers { permanent_uuid: "8202526897ed4dd9ac5c03333b95bc18" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 45191 } } peers { permanent_uuid: "46841f5ad9af4fb99fc13a8d3318db4c" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 33569 } } }
I20260619 14:34:39.779342 15609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8202526897ed4dd9ac5c03333b95bc18 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 14:34:39.779333 15621 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6d33269977be4859a11e98dacc85ad23: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:34:39.779449 15621 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:34:39.779660 15607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d33269977be4859a11e98dacc85ad23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d33269977be4859a11e98dacc85ad23" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41727 } } peers { permanent_uuid: "8202526897ed4dd9ac5c03333b95bc18" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 45191 } } peers { permanent_uuid: "46841f5ad9af4fb99fc13a8d3318db4c" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 33569 } } }
I20260619 14:34:39.779783 15607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d33269977be4859a11e98dacc85ad23 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:34:39.780089 15598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46841f5ad9af4fb99fc13a8d3318db4c: Bootstrap starting.
I20260619 14:34:39.776947 15379 master_runner.cc:186] Error getting master registration for 127.2.2.60:33569: OK, instance_id { permanent_uuid: "46841f5ad9af4fb99fc13a8d3318db4c" instance_seqno: 1781879679717129 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 14:34:39.780422 15379 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 14:34:39.782866 15598 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46841f5ad9af4fb99fc13a8d3318db4c: Neither blocks nor log segments found. Creating new log.
I20260619 14:34:39.784147 15598 log.cc:824] T 00000000000000000000000000000000 P 46841f5ad9af4fb99fc13a8d3318db4c: Log is configured to *not* fsync() on all Append() calls
I20260619 14:34:39.782683 15454 master_runner.cc:186] Error getting master registration for 127.2.2.60:33569: OK, instance_id { permanent_uuid: "46841f5ad9af4fb99fc13a8d3318db4c" instance_seqno: 1781879679717129 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 14:34:39.786509 15454 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 14:34:39.786602 15598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46841f5ad9af4fb99fc13a8d3318db4c: No bootstrap required, opened a new log
I20260619 14:34:39.791683 15598 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46841f5ad9af4fb99fc13a8d3318db4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d33269977be4859a11e98dacc85ad23" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41727 } } peers { permanent_uuid: "8202526897ed4dd9ac5c03333b95bc18" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 45191 } } peers { permanent_uuid: "46841f5ad9af4fb99fc13a8d3318db4c" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 33569 } }
I20260619 14:34:39.792032 15598 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46841f5ad9af4fb99fc13a8d3318db4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:34:39.792111 15598 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46841f5ad9af4fb99fc13a8d3318db4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46841f5ad9af4fb99fc13a8d3318db4c, State: Initialized, Role: FOLLOWER
I20260619 14:34:39.792721 15598 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46841f5ad9af4fb99fc13a8d3318db4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d33269977be4859a11e98dacc85ad23" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41727 } } peers { permanent_uuid: "8202526897ed4dd9ac5c03333b95bc18" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 45191 } } peers { permanent_uuid: "46841f5ad9af4fb99fc13a8d3318db4c" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 33569 } }
I20260619 14:34:39.793285 15642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46841f5ad9af4fb99fc13a8d3318db4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d33269977be4859a11e98dacc85ad23" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41727 } } peers { permanent_uuid: "8202526897ed4dd9ac5c03333b95bc18" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 45191 } } peers { permanent_uuid: "46841f5ad9af4fb99fc13a8d3318db4c" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 33569 } } }
I20260619 14:34:39.793469 15642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46841f5ad9af4fb99fc13a8d3318db4c [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:34:39.796865 15527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:39.782609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56700 (local address 127.2.2.61:45191)
0619 14:34:39.782723 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 14:34:39.782729 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:34:39.787073 (+ 4344us) server_negotiation.cc:407] Connection header received
0619 14:34:39.787213 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:34:39.787216 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:34:39.787270 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:34:39.787366 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:34:39.788335 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:39.789150 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:39.790533 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:39.790886 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:39.791004 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:39.792930 (+ 1926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:34:39.792934 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:34:39.792946 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:34:39.792975 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:34:39.794904 (+ 1929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:39.795457 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:39.795461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:39.795463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:39.795554 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:39.795883 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:39.795886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:39.795888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:39.796131 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:34:39.796273 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 14:34:39.796575 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 14:34:39.796702 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 14:34:39.802433 15598 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46841f5ad9af4fb99fc13a8d3318db4c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:34:39.808492 15604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:39.786657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38640 (local address 127.2.2.60:33569)
0619 14:34:39.786776 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 14:34:39.786781 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:34:39.796963 (+ 10182us) server_negotiation.cc:407] Connection header received
0619 14:34:39.797163 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:34:39.797167 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:34:39.797227 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:34:39.797284 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:34:39.798279 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:39.799198 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:39.800474 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:39.800825 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:39.800933 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:39.802821 (+ 1888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:34:39.802826 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:34:39.802829 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:34:39.802859 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:34:39.805715 (+ 2856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:39.806285 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:39.806292 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:39.806294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:39.806411 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:39.807337 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:39.807341 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:39.807343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:39.807596 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:34:39.807781 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0619 14:34:39.808154 (+ 373us) server_negotiation.cc:299] Negotiation successful
0619 14:34:39.808291 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 14:34:39.812319 15529 master_runner.cc:263] No leader master found from master 46841f5ad9af4fb99fc13a8d3318db4c
I20260619 14:34:39.812407 15529 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 14:34:39.813098 15450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:39.783046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52256 (local address 127.2.2.62:41727)
0619 14:34:39.783164 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 14:34:39.783169 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:34:39.784828 (+ 1659us) server_negotiation.cc:407] Connection header received
0619 14:34:39.802782 (+ 17954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:34:39.802786 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:34:39.802851 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:34:39.802956 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:34:39.803912 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:39.804799 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:39.806634 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:39.806959 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:39.807062 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:39.808974 (+ 1912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:34:39.808978 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:34:39.808981 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:34:39.809008 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:34:39.810813 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:39.811337 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:39.811342 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:39.811344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:39.811454 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:39.811801 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:39.811805 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:39.811807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:39.812040 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:34:39.812187 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 14:34:39.812865 (+ 678us) server_negotiation.cc:299] Negotiation successful
0619 14:34:39.812977 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260619 14:34:39.817368 15653 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 46841f5ad9af4fb99fc13a8d3318db4c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:34:39.817481 15653 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:34:39.953616 15607 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d33269977be4859a11e98dacc85ad23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:34:39.953845 15607 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d33269977be4859a11e98dacc85ad23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d33269977be4859a11e98dacc85ad23" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41727 } } peers { permanent_uuid: "8202526897ed4dd9ac5c03333b95bc18" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 45191 } } peers { permanent_uuid: "46841f5ad9af4fb99fc13a8d3318db4c" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 33569 } }
I20260619 14:34:39.955011 15607 leader_election.cc:288] T 00000000000000000000000000000000 P 6d33269977be4859a11e98dacc85ad23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8202526897ed4dd9ac5c03333b95bc18 (127.2.2.61:45191), 46841f5ad9af4fb99fc13a8d3318db4c (127.2.2.60:33569)
I20260619 14:34:39.955832 15495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6d33269977be4859a11e98dacc85ad23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8202526897ed4dd9ac5c03333b95bc18" is_pre_election: true
I20260619 14:34:39.956202 15495 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8202526897ed4dd9ac5c03333b95bc18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d33269977be4859a11e98dacc85ad23 in term 0.
I20260619 14:34:39.957077 15572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6d33269977be4859a11e98dacc85ad23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46841f5ad9af4fb99fc13a8d3318db4c" is_pre_election: true
I20260619 14:34:39.957398 15572 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 46841f5ad9af4fb99fc13a8d3318db4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d33269977be4859a11e98dacc85ad23 in term 0.
I20260619 14:34:39.957934 15397 leader_election.cc:302] T 00000000000000000000000000000000 P 6d33269977be4859a11e98dacc85ad23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46841f5ad9af4fb99fc13a8d3318db4c, 6d33269977be4859a11e98dacc85ad23; no voters:
I20260619 14:34:39.958506 15607 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d33269977be4859a11e98dacc85ad23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:34:39.958599 15607 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d33269977be4859a11e98dacc85ad23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:34:39.958647 15607 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d33269977be4859a11e98dacc85ad23 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:34:39.959980 15607 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d33269977be4859a11e98dacc85ad23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d33269977be4859a11e98dacc85ad23" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41727 } } peers { permanent_uuid: "8202526897ed4dd9ac5c03333b95bc18" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 45191 } } peers { permanent_uuid: "46841f5ad9af4fb99fc13a8d3318db4c" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 33569 } }
I20260619 14:34:39.960405 15607 leader_election.cc:288] T 00000000000000000000000000000000 P 6d33269977be4859a11e98dacc85ad23 [CANDIDATE]: Term 1 election: Requested vote from peers 8202526897ed4dd9ac5c03333b95bc18 (127.2.2.61:45191), 46841f5ad9af4fb99fc13a8d3318db4c (127.2.2.60:33569)
I20260619 14:34:39.960834 15495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6d33269977be4859a11e98dacc85ad23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8202526897ed4dd9ac5c03333b95bc18"
I20260619 14:34:39.960970 15495 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8202526897ed4dd9ac5c03333b95bc18 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:34:39.962278 15495 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8202526897ed4dd9ac5c03333b95bc18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d33269977be4859a11e98dacc85ad23 in term 1.
I20260619 14:34:39.962901 15572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6d33269977be4859a11e98dacc85ad23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46841f5ad9af4fb99fc13a8d3318db4c"
I20260619 14:34:39.963037 15572 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46841f5ad9af4fb99fc13a8d3318db4c [term 0 FOLLOWER]: Advancing to term 1
W20260619 14:34:39.963414 15551 tablet.cc:2369] T 00000000000000000000000000000000 P 46841f5ad9af4fb99fc13a8d3318db4c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:34:39.964205 15572 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 46841f5ad9af4fb99fc13a8d3318db4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d33269977be4859a11e98dacc85ad23 in term 1.
I20260619 14:34:39.964555 15397 leader_election.cc:302] T 00000000000000000000000000000000 P 6d33269977be4859a11e98dacc85ad23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46841f5ad9af4fb99fc13a8d3318db4c, 6d33269977be4859a11e98dacc85ad23; no voters:
I20260619 14:34:39.964839 15607 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d33269977be4859a11e98dacc85ad23 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:34:39.965072 15607 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d33269977be4859a11e98dacc85ad23 [term 1 LEADER]: Becoming Leader. State: Replica: 6d33269977be4859a11e98dacc85ad23, State: Running, Role: LEADER
I20260619 14:34:39.965410 15607 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d33269977be4859a11e98dacc85ad23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d33269977be4859a11e98dacc85ad23" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41727 } } peers { permanent_uuid: "8202526897ed4dd9ac5c03333b95bc18" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 45191 } } peers { permanent_uuid: "46841f5ad9af4fb99fc13a8d3318db4c" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 33569 } }
I20260619 14:34:39.967510 15607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d33269977be4859a11e98dacc85ad23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d33269977be4859a11e98dacc85ad23. Latest consensus state: current_term: 1 leader_uuid: "6d33269977be4859a11e98dacc85ad23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d33269977be4859a11e98dacc85ad23" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41727 } } peers { permanent_uuid: "8202526897ed4dd9ac5c03333b95bc18" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 45191 } } peers { permanent_uuid: "46841f5ad9af4fb99fc13a8d3318db4c" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 33569 } } }
I20260619 14:34:39.967680 15607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d33269977be4859a11e98dacc85ad23 [sys.catalog]: This master's current role is: LEADER
I20260619 14:34:39.971411 15657 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:34:39.972339 15657 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260619 14:34:39.977948 15475 tablet.cc:2369] T 00000000000000000000000000000000 P 8202526897ed4dd9ac5c03333b95bc18: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:34:39.979945 15572 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 46841f5ad9af4fb99fc13a8d3318db4c [term 1 FOLLOWER]: Refusing update from remote peer 6d33269977be4859a11e98dacc85ad23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:34:39.980679 15655 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6d33269977be4859a11e98dacc85ad23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46841f5ad9af4fb99fc13a8d3318db4c" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 33569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:34:39.984759 15642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46841f5ad9af4fb99fc13a8d3318db4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d33269977be4859a11e98dacc85ad23. Latest consensus state: current_term: 1 leader_uuid: "6d33269977be4859a11e98dacc85ad23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d33269977be4859a11e98dacc85ad23" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41727 } } peers { permanent_uuid: "8202526897ed4dd9ac5c03333b95bc18" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 45191 } } peers { permanent_uuid: "46841f5ad9af4fb99fc13a8d3318db4c" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 33569 } } }
I20260619 14:34:39.984901 15642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46841f5ad9af4fb99fc13a8d3318db4c [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:34:39.985993 15495 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8202526897ed4dd9ac5c03333b95bc18 [term 1 FOLLOWER]: Refusing update from remote peer 6d33269977be4859a11e98dacc85ad23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:34:39.986675 15655 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6d33269977be4859a11e98dacc85ad23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8202526897ed4dd9ac5c03333b95bc18" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 45191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:34:39.990829 15609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8202526897ed4dd9ac5c03333b95bc18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d33269977be4859a11e98dacc85ad23. Latest consensus state: current_term: 1 leader_uuid: "6d33269977be4859a11e98dacc85ad23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d33269977be4859a11e98dacc85ad23" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41727 } } peers { permanent_uuid: "8202526897ed4dd9ac5c03333b95bc18" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 45191 } } peers { permanent_uuid: "46841f5ad9af4fb99fc13a8d3318db4c" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 33569 } } }
I20260619 14:34:39.990981 15609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8202526897ed4dd9ac5c03333b95bc18 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:34:39.994270 15655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d33269977be4859a11e98dacc85ad23 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6d33269977be4859a11e98dacc85ad23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d33269977be4859a11e98dacc85ad23" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41727 } } peers { permanent_uuid: "8202526897ed4dd9ac5c03333b95bc18" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 45191 } } peers { permanent_uuid: "46841f5ad9af4fb99fc13a8d3318db4c" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 33569 } } }
I20260619 14:34:39.994407 15655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d33269977be4859a11e98dacc85ad23 [sys.catalog]: This master's current role is: LEADER
I20260619 14:34:39.994755 15642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46841f5ad9af4fb99fc13a8d3318db4c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6d33269977be4859a11e98dacc85ad23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d33269977be4859a11e98dacc85ad23" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41727 } } peers { permanent_uuid: "8202526897ed4dd9ac5c03333b95bc18" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 45191 } } peers { permanent_uuid: "46841f5ad9af4fb99fc13a8d3318db4c" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 33569 } } }
I20260619 14:34:39.994871 15642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46841f5ad9af4fb99fc13a8d3318db4c [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:34:39.996686 15655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d33269977be4859a11e98dacc85ad23 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6d33269977be4859a11e98dacc85ad23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d33269977be4859a11e98dacc85ad23" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41727 } } peers { permanent_uuid: "8202526897ed4dd9ac5c03333b95bc18" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 45191 } } peers { permanent_uuid: "46841f5ad9af4fb99fc13a8d3318db4c" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 33569 } } }
I20260619 14:34:39.996803 15655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d33269977be4859a11e98dacc85ad23 [sys.catalog]: This master's current role is: LEADER
I20260619 14:34:39.997611 15609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8202526897ed4dd9ac5c03333b95bc18 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6d33269977be4859a11e98dacc85ad23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d33269977be4859a11e98dacc85ad23" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41727 } } peers { permanent_uuid: "8202526897ed4dd9ac5c03333b95bc18" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 45191 } } peers { permanent_uuid: "46841f5ad9af4fb99fc13a8d3318db4c" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 33569 } } }
I20260619 14:34:39.997735 15609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8202526897ed4dd9ac5c03333b95bc18 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:34:39.999008 15657 catalog_manager.cc:1347] Generated new cluster ID: 01f06ac3ad164a03af41896ca4debc24
I20260619 14:34:39.999074 15657 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:34:40.014147 15657 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:34:40.015712 15657 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:34:40.028331 15657 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d33269977be4859a11e98dacc85ad23: Generated new TSK 0
I20260619 14:34:40.029325 15657 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:34:40.780324 15631 catalog_manager.cc:1259] Loaded cluster ID: 01f06ac3ad164a03af41896ca4debc24
I20260619 14:34:40.780433 15631 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8202526897ed4dd9ac5c03333b95bc18: loading cluster ID for follower catalog manager: success
I20260619 14:34:40.782299 15631 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8202526897ed4dd9ac5c03333b95bc18: acquiring CA information for follower catalog manager: success
I20260619 14:34:40.783594 15631 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8202526897ed4dd9ac5c03333b95bc18: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 14:34:40.818815 15653 catalog_manager.cc:1259] Loaded cluster ID: 01f06ac3ad164a03af41896ca4debc24
I20260619 14:34:40.818903 15653 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 46841f5ad9af4fb99fc13a8d3318db4c: loading cluster ID for follower catalog manager: success
I20260619 14:34:40.820696 15653 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 46841f5ad9af4fb99fc13a8d3318db4c: acquiring CA information for follower catalog manager: success
I20260619 14:34:40.821883 15653 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 46841f5ad9af4fb99fc13a8d3318db4c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 14:34:41.100957 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 15379
I20260619 14:34:41.107223 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 15454
I20260619 14:34:41.113016 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 15529
2026-06-19T14:34:41Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3185 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781879514993456-2056-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:34:41 dist-test-slave-70rm krb5kdc[15672](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:34:41 dist-test-slave-70rm krb5kdc[15672](info): set up 2 sockets
Jun 19 14:34:41 dist-test-slave-70rm krb5kdc[15672](info): commencing operation
krb5kdc: starting...
W20260619 14:34:42.162403 2056 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Jun 19 14:34:42 dist-test-slave-70rm krb5kdc[15672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:34:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:34:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.2.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab.
Entry for principal kudu/127.2.2.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab.
I20260619 14:34:42.284919 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781879514993456-2056-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781879514993456-2056-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781879514993456-2056-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:37447
--webserver_interface=127.2.2.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:40621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.2.62:37447
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:34:42.396539 15685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:34:42.396831 15685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:34:42.396872 15685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:34:42.400810 15685 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:34:42.400863 15685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:34:42.400883 15685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:34:42.400900 15685 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:34:42.400915 15685 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:34:42.406080 15685 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:40621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781879514993456-2056-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781879514993456-2056-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.2.62:37447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:37447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.2.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781879514993456-2056-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:34:42.407094 15685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:34:42.408049 15685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:34:42.416677 15693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:34:42.416677 15690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:34:42.416689 15691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:34:42.416988 15685 server_base.cc:1034] running on GCE node
I20260619 14:34:42.417740 15685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:34:42.418998 15685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:34:42.420176 15685 hybrid_clock.cc:648] HybridClock initialized: now 1781879682420159 us; error 31 us; skew 500 ppm
Jun 19 14:34:42 dist-test-slave-70rm krb5kdc[15672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879682, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:34:42.423038 15685 init.cc:377] Logged in from keytab as kudu/127.2.2.62@KRBTEST.COM (short username kudu)
I20260619 14:34:42.423897 15685 webserver.cc:460] Webserver started at http://127.2.2.62:37789/ using document root <none> and password file <none>
I20260619 14:34:42.424443 15685 fs_manager.cc:359] Metadata directory not provided
I20260619 14:34:42.424501 15685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:34:42.424666 15685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:34:42.426800 15685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781879514993456-2056-0/minicluster-data/master-0/data/instance:
uuid: "00506259ae4a4d6cbf9f07ab87be71ce"
format_stamp: "Formatted at 2026-06-19 14:34:42 on dist-test-slave-70rm"
server_key: "1903328bdc4f538c3bdac17eeea38efd"
server_key_iv: "756765ff4fdaca1fb3b08e22ed3ca664"
server_key_version: "encryptionkey@0"
I20260619 14:34:42.427341 15685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781879514993456-2056-0/minicluster-data/master-0/wal/instance:
uuid: "00506259ae4a4d6cbf9f07ab87be71ce"
format_stamp: "Formatted at 2026-06-19 14:34:42 on dist-test-slave-70rm"
server_key: "1903328bdc4f538c3bdac17eeea38efd"
server_key_iv: "756765ff4fdaca1fb3b08e22ed3ca664"
server_key_version: "encryptionkey@0"
I20260619 14:34:42.431216 15685 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:34:42.433625 15700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:34:42.434973 15685 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:34:42.435117 15685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781879514993456-2056-0/minicluster-data/master-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781879514993456-2056-0/minicluster-data/master-0/wal
uuid: "00506259ae4a4d6cbf9f07ab87be71ce"
format_stamp: "Formatted at 2026-06-19 14:34:42 on dist-test-slave-70rm"
server_key: "1903328bdc4f538c3bdac17eeea38efd"
server_key_iv: "756765ff4fdaca1fb3b08e22ed3ca664"
server_key_version: "encryptionkey@0"
I20260619 14:34:42.435236 15685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781879514993456-2056-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781879514993456-2056-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781879514993456-2056-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:34:42.466203 15685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:34:42.469225 15685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:34:42.469436 15685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:34:42.476787 15685 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.62:37447
I20260619 14:34:42.476811 15752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.62:37447 every 8 connection(s)
I20260619 14:34:42.477787 15685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
I20260619 14:34:42.480865 15753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:34:42.481601 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 15685
I20260619 14:34:42.481737 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781879514993456-2056-0/minicluster-data/master-0/wal/instance
I20260619 14:34:42.482084 2056 external_mini_cluster.cc:1428] Setting key 332918a1f66579a611f0eb54c489a4d7
I20260619 14:34:42.488114 15753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00506259ae4a4d6cbf9f07ab87be71ce: Bootstrap starting.
I20260619 14:34:42.490636 15753 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00506259ae4a4d6cbf9f07ab87be71ce: Neither blocks nor log segments found. Creating new log.
Jun 19 14:34:42 dist-test-slave-70rm krb5kdc[15672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:34:42.491456 15753 log.cc:824] T 00000000000000000000000000000000 P 00506259ae4a4d6cbf9f07ab87be71ce: Log is configured to *not* fsync() on all Append() calls
I20260619 14:34:42.493464 15753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00506259ae4a4d6cbf9f07ab87be71ce: No bootstrap required, opened a new log
I20260619 14:34:42.496796 15756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:42.483609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38398 (local address 127.2.2.62:37447)
0619 14:34:42.484083 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0619 14:34:42.484099 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 14:34:42.484136 (+ 37us) server_negotiation.cc:407] Connection header received
0619 14:34:42.484820 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:34:42.484837 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:34:42.485185 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:34:42.485549 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:34:42.486437 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:42.487437 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:42.488382 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:42.488883 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:42.489058 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:42.491767 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:34:42.491774 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:34:42.491794 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:34:42.491834 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:34:42.493690 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:42.494287 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:42.494299 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:42.494305 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:42.494440 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:42.494761 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:42.494763 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:42.494764 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:42.495201 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:34:42.495372 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 14:34:42.495695 (+ 323us) server_negotiation.cc:299] Negotiation successful
0619 14:34:42.496035 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":154,"threads_started":1}
I20260619 14:34:42.497627 15753 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00506259ae4a4d6cbf9f07ab87be71ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00506259ae4a4d6cbf9f07ab87be71ce" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 37447 } }
I20260619 14:34:42.497892 15753 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00506259ae4a4d6cbf9f07ab87be71ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:34:42.497956 15753 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00506259ae4a4d6cbf9f07ab87be71ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00506259ae4a4d6cbf9f07ab87be71ce, State: Initialized, Role: FOLLOWER
I20260619 14:34:42.498453 15753 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00506259ae4a4d6cbf9f07ab87be71ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00506259ae4a4d6cbf9f07ab87be71ce" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 37447 } }
I20260619 14:34:42.498610 15753 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00506259ae4a4d6cbf9f07ab87be71ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:34:42.498678 15753 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00506259ae4a4d6cbf9f07ab87be71ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:34:42.498790 15753 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00506259ae4a4d6cbf9f07ab87be71ce [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:34:42.499857 2056 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:34:42.499977 15753 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00506259ae4a4d6cbf9f07ab87be71ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00506259ae4a4d6cbf9f07ab87be71ce" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 37447 } }
I20260619 14:34:42.500321 15753 leader_election.cc:302] T 00000000000000000000000000000000 P 00506259ae4a4d6cbf9f07ab87be71ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00506259ae4a4d6cbf9f07ab87be71ce; no voters:
I20260619 14:34:42.500672 15753 leader_election.cc:288] T 00000000000000000000000000000000 P 00506259ae4a4d6cbf9f07ab87be71ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:34:42.500792 15758 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00506259ae4a4d6cbf9f07ab87be71ce [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:34:42.500998 15758 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00506259ae4a4d6cbf9f07ab87be71ce [term 1 LEADER]: Becoming Leader. State: Replica: 00506259ae4a4d6cbf9f07ab87be71ce, State: Running, Role: LEADER
I20260619 14:34:42.501391 15758 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00506259ae4a4d6cbf9f07ab87be71ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00506259ae4a4d6cbf9f07ab87be71ce" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 37447 } }
I20260619 14:34:42.501967 15753 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00506259ae4a4d6cbf9f07ab87be71ce [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:34:42.503191 15759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00506259ae4a4d6cbf9f07ab87be71ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00506259ae4a4d6cbf9f07ab87be71ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00506259ae4a4d6cbf9f07ab87be71ce" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 37447 } } }
I20260619 14:34:42.503197 15760 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00506259ae4a4d6cbf9f07ab87be71ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00506259ae4a4d6cbf9f07ab87be71ce. Latest consensus state: current_term: 1 leader_uuid: "00506259ae4a4d6cbf9f07ab87be71ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00506259ae4a4d6cbf9f07ab87be71ce" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 37447 } } }
I20260619 14:34:42.503368 15760 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00506259ae4a4d6cbf9f07ab87be71ce [sys.catalog]: This master's current role is: LEADER
I20260619 14:34:42.503710 15759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00506259ae4a4d6cbf9f07ab87be71ce [sys.catalog]: This master's current role is: LEADER
W20260619 14:34:42.506604 15773 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 00506259ae4a4d6cbf9f07ab87be71ce: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:34:42.506687 15773 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:34:42.506755 15768 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:34:42.507401 15768 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:34:42.513213 15768 catalog_manager.cc:1347] Generated new cluster ID: 39488580556349549586676cbaa1acf3
I20260619 14:34:42.513298 15768 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:34:42.532493 15768 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:34:42.533469 15768 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:34:42.537752 15768 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00506259ae4a4d6cbf9f07ab87be71ce: Generated new TSK 0
I20260619 14:34:42.538499 15768 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:34:42.556384 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 15685
2026-06-19T14:34:42Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1441 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:34:42 dist-test-slave-70rm krb5kdc[15780](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:34:42 dist-test-slave-70rm krb5kdc[15780](info): set up 2 sockets
Jun 19 14:34:42 dist-test-slave-70rm krb5kdc[15780](info): commencing operation
krb5kdc: starting...
W20260619 14:34:43.591248 2056 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Jun 19 14:34:43 dist-test-slave-70rm krb5kdc[15780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879683, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:34:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:34:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.2.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab.
Entry for principal kudu/127.2.2.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab.
I20260619 14:34:43.714975 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:46797
--webserver_interface=127.2.2.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:39547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.2.62:46797
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:34:43.827287 15793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:34:43.827591 15793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:34:43.827633 15793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:34:43.831810 15793 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:34:43.831875 15793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:34:43.831895 15793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:34:43.831912 15793 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:34:43.831928 15793 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:34:43.837312 15793 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:39547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.2.62:46797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:46797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.2.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:34:43.838508 15793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:34:43.839576 15793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:34:43.848543 15801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:34:43.848592 15799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:34:43.848606 15798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:34:43.848683 15793 server_base.cc:1034] running on GCE node
I20260619 14:34:43.849427 15793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:34:43.850714 15793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:34:43.851889 15793 hybrid_clock.cc:648] HybridClock initialized: now 1781879683851854 us; error 51 us; skew 500 ppm
Jun 19 14:34:43 dist-test-slave-70rm krb5kdc[15780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879683, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:34:43.862250 15793 init.cc:377] Logged in from keytab as kudu/127.2.2.62@KRBTEST.COM (short username kudu)
I20260619 14:34:43.863291 15793 webserver.cc:460] Webserver started at http://127.2.2.62:45891/ using document root <none> and password file <none>
I20260619 14:34:43.863881 15793 fs_manager.cc:359] Metadata directory not provided
I20260619 14:34:43.863945 15793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:34:43.864192 15793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:34:43.866415 15793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/master-0/data/instance:
uuid: "f971ee13217e41268e22d92d2a111986"
format_stamp: "Formatted at 2026-06-19 14:34:43 on dist-test-slave-70rm"
server_key: "4e56bd85d38ea569d4be8f50d6aaa9ad"
server_key_iv: "d022057fa1f55ec60b11a6bb14e89137"
server_key_version: "encryptionkey@0"
I20260619 14:34:43.867002 15793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/master-0/wal/instance:
uuid: "f971ee13217e41268e22d92d2a111986"
format_stamp: "Formatted at 2026-06-19 14:34:43 on dist-test-slave-70rm"
server_key: "4e56bd85d38ea569d4be8f50d6aaa9ad"
server_key_iv: "d022057fa1f55ec60b11a6bb14e89137"
server_key_version: "encryptionkey@0"
I20260619 14:34:43.870849 15793 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:34:43.873252 15808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:34:43.874593 15793 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:34:43.874738 15793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/master-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/master-0/wal
uuid: "f971ee13217e41268e22d92d2a111986"
format_stamp: "Formatted at 2026-06-19 14:34:43 on dist-test-slave-70rm"
server_key: "4e56bd85d38ea569d4be8f50d6aaa9ad"
server_key_iv: "d022057fa1f55ec60b11a6bb14e89137"
server_key_version: "encryptionkey@0"
I20260619 14:34:43.874854 15793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:34:43.901516 15793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:34:43.904959 15793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:34:43.905180 15793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:34:43.912575 15793 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.62:46797
I20260619 14:34:43.912860 15860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.62:46797 every 8 connection(s)
I20260619 14:34:43.913633 15793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
I20260619 14:34:43.916451 15861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:34:43.921849 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 15793
I20260619 14:34:43.922022 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/master-0/wal/instance
I20260619 14:34:43.922366 2056 external_mini_cluster.cc:1428] Setting key 647c97aff9a48f43fe94a57afc808387
I20260619 14:34:43.923009 15861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986: Bootstrap starting.
I20260619 14:34:43.925508 15861 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986: Neither blocks nor log segments found. Creating new log.
I20260619 14:34:43.926376 15861 log.cc:824] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986: Log is configured to *not* fsync() on all Append() calls
I20260619 14:34:43.928674 15861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986: No bootstrap required, opened a new log
Jun 19 14:34:43 dist-test-slave-70rm krb5kdc[15780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879683, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:34:43.933691 15861 raft_consensus.cc:348] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f971ee13217e41268e22d92d2a111986" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 46797 } }
I20260619 14:34:43.933980 15861 raft_consensus.cc:374] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:34:43.934056 15861 raft_consensus.cc:729] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f971ee13217e41268e22d92d2a111986, State: Initialized, Role: FOLLOWER
I20260619 14:34:43.934652 15861 consensus_queue.cc:260] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f971ee13217e41268e22d92d2a111986" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 46797 } }
I20260619 14:34:43.934788 15861 raft_consensus.cc:388] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:34:43.934890 15861 raft_consensus.cc:482] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:34:43.935029 15861 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:34:43.936283 15861 raft_consensus.cc:504] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f971ee13217e41268e22d92d2a111986" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 46797 } }
I20260619 14:34:43.936678 15861 leader_election.cc:302] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f971ee13217e41268e22d92d2a111986; no voters:
I20260619 14:34:43.937057 15861 leader_election.cc:288] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:34:43.937137 15866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:34:43.937397 15866 raft_consensus.cc:686] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986 [term 1 LEADER]: Becoming Leader. State: Replica: f971ee13217e41268e22d92d2a111986, State: Running, Role: LEADER
I20260619 14:34:43.937951 15866 consensus_queue.cc:237] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f971ee13217e41268e22d92d2a111986" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 46797 } }
I20260619 14:34:43.938602 15861 sys_catalog.cc:564] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:34:43.938652 15864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:43.923939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38856 (local address 127.2.2.62:46797)
0619 14:34:43.924366 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0619 14:34:43.924386 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 14:34:43.924437 (+ 51us) server_negotiation.cc:407] Connection header received
0619 14:34:43.925069 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:34:43.925085 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:34:43.925426 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:34:43.925780 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:34:43.926642 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:43.928233 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:43.929199 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:43.929945 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:43.930113 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:43.933181 (+ 3068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:34:43.933187 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:34:43.933202 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:34:43.933230 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:34:43.935445 (+ 2215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:43.935976 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:43.935981 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:43.935986 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:43.936101 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:43.936452 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:43.936456 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:43.936457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:43.936940 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:34:43.937162 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0619 14:34:43.937470 (+ 308us) server_negotiation.cc:299] Negotiation successful
0619 14:34:43.937767 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":164,"threads_started":1}
I20260619 14:34:43.939265 15866 sys_catalog.cc:455] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f971ee13217e41268e22d92d2a111986. Latest consensus state: current_term: 1 leader_uuid: "f971ee13217e41268e22d92d2a111986" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f971ee13217e41268e22d92d2a111986" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 46797 } } }
I20260619 14:34:43.939404 15866 sys_catalog.cc:458] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986 [sys.catalog]: This master's current role is: LEADER
I20260619 14:34:43.939625 15866 sys_catalog.cc:455] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f971ee13217e41268e22d92d2a111986" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f971ee13217e41268e22d92d2a111986" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 46797 } } }
I20260619 14:34:43.939724 15866 sys_catalog.cc:458] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986 [sys.catalog]: This master's current role is: LEADER
W20260619 14:34:43.943521 15881 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:34:43.943624 15881 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:34:43.943688 15872 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:34:43.944684 15872 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:34:43.950786 15872 catalog_manager.cc:1347] Generated new cluster ID: f623e242ae6241d69aa547d8601a075c
I20260619 14:34:43.950901 15872 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:34:43.965549 15872 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:34:43.967023 15872 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:34:43.973675 15872 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986: Generated new TSK 0
I20260619 14:34:43.974545 15872 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.2.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab.
Entry for principal kudu/127.2.2.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab.
I20260619 14:34:44.010084 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.1:0
--local_ip_for_outbound_sockets=127.2.2.1
--webserver_interface=127.2.2.1
--webserver_port=0
--tserver_master_addrs=127.2.2.62:46797
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab
--principal=kudu/127.2.2.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:39547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:34:44.136055 15887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:34:44.136333 15887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:34:44.136421 15887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:34:44.140583 15887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:34:44.140650 15887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:34:44.140753 15887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.1
I20260619 14:34:44.146090 15887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:39547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab
--principal=kudu/127.2.2.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.2.1
--webserver_port=0
--tserver_master_addrs=127.2.2.62:46797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.1
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:34:44.147228 15887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:34:44.148283 15887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:34:44.158210 15892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:34:44.158170 15895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:34:44.158356 15887 server_base.cc:1034] running on GCE node
W20260619 14:34:44.158210 15893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:34:44.158984 15887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:34:44.159634 15887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:34:44.160786 15887 hybrid_clock.cc:648] HybridClock initialized: now 1781879684160754 us; error 56 us; skew 500 ppm
Jun 19 14:34:44 dist-test-slave-70rm krb5kdc[15780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879684, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:34:44.163666 15887 init.cc:377] Logged in from keytab as kudu/127.2.2.1@KRBTEST.COM (short username kudu)
I20260619 14:34:44.164528 15887 webserver.cc:460] Webserver started at http://127.2.2.1:38943/ using document root <none> and password file <none>
I20260619 14:34:44.165060 15887 fs_manager.cc:359] Metadata directory not provided
I20260619 14:34:44.165119 15887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:34:44.165294 15887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:34:44.167452 15887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-0/data/instance:
uuid: "c71ce75c24df481097464fb1957b76b4"
format_stamp: "Formatted at 2026-06-19 14:34:44 on dist-test-slave-70rm"
server_key: "a51a61d3af2563badf3e5fbec931e7cc"
server_key_iv: "cdb1c3a55a66061db5e6c614947ba594"
server_key_version: "encryptionkey@0"
I20260619 14:34:44.167970 15887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-0/wal/instance:
uuid: "c71ce75c24df481097464fb1957b76b4"
format_stamp: "Formatted at 2026-06-19 14:34:44 on dist-test-slave-70rm"
server_key: "a51a61d3af2563badf3e5fbec931e7cc"
server_key_iv: "cdb1c3a55a66061db5e6c614947ba594"
server_key_version: "encryptionkey@0"
I20260619 14:34:44.171811 15887 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:34:44.174326 15902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:34:44.175583 15887 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:34:44.175722 15887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-0/wal
uuid: "c71ce75c24df481097464fb1957b76b4"
format_stamp: "Formatted at 2026-06-19 14:34:44 on dist-test-slave-70rm"
server_key: "a51a61d3af2563badf3e5fbec931e7cc"
server_key_iv: "cdb1c3a55a66061db5e6c614947ba594"
server_key_version: "encryptionkey@0"
I20260619 14:34:44.175840 15887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:34:44.218688 15887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:34:44.222162 15887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:34:44.222376 15887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:34:44.222985 15887 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:34:44.223822 15887 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:34:44.223891 15887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:34:44.223960 15887 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:34:44.224010 15887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:34:44.233206 15887 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.1:46097
I20260619 14:34:44.233235 16015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.1:46097 every 8 connection(s)
I20260619 14:34:44.234228 15887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
I20260619 14:34:44.237675 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 15887
I20260619 14:34:44.237826 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-0/wal/instance
I20260619 14:34:44.238109 2056 external_mini_cluster.cc:1428] Setting key 8f304bf9850f4990f5147594e31bcde6
Jun 19 14:34:44 dist-test-slave-70rm krb5kdc[15780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879684, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.1@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.2.2@KRBTEST.COM; defaulting to no policy
I20260619 14:34:44.249749 15864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:44.236100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:40245 (local address 127.2.2.62:46797)
0619 14:34:44.236236 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 14:34:44.236240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:34:44.237256 (+ 1016us) server_negotiation.cc:407] Connection header received
0619 14:34:44.238344 (+ 1088us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:34:44.238349 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:34:44.238409 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:34:44.238507 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:34:44.239981 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:44.240804 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:44.242652 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:44.242880 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:44.242962 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:44.245924 (+ 2962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:34:44.245928 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:34:44.245931 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:34:44.245960 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:34:44.247561 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:44.248162 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:44.248166 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:44.248168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:44.248252 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:44.248600 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:44.248605 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:44.248607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:44.248795 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:34:44.248901 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 14:34:44.249443 (+ 542us) server_negotiation.cc:299] Negotiation successful
0619 14:34:44.249579 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.2@KRBTEST.COM" created.
I20260619 14:34:44.250785 16018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:44.236441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46797 (local address 127.2.2.1:40245)
0619 14:34:44.237119 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:44.237162 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 14:34:44.238070 (+ 908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:34:44.238735 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:34:44.238744 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:34:44.239122 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:34:44.239794 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:44.239812 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:44.240953 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:44.240956 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:44.242524 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:44.242540 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:44.242990 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:44.243055 (+ 65us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:44.243308 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:44.243975 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:34:44.244004 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:34:44.245749 (+ 1745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:34:44.247687 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:34:44.247694 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:34:44.247711 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:34:44.248059 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:34:44.248344 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:34:44.248348 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:34:44.248350 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:34:44.248499 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:34:44.248900 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:34:44.248907 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:34:44.249195 (+ 288us) client_negotiation.cc:769] Sending connection context
0619 14:34:44.249394 (+ 199us) client_negotiation.cc:240] Negotiation successful
0619 14:34:44.249760 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":113,"threads_started":1}
I20260619 14:34:44.252148 16016 heartbeater.cc:347] Connected to a master server at 127.2.2.62:46797
I20260619 14:34:44.252424 16016 heartbeater.cc:464] Registering TS with master...
I20260619 14:34:44.252947 16016 heartbeater.cc:511] Master 127.2.2.62:46797 requested a full tablet report, sending...
I20260619 14:34:44.254669 15825 ts_manager.cc:194] Registered new tserver with Master: c71ce75c24df481097464fb1957b76b4 (127.2.2.1:46097)
I20260619 14:34:44.255810 15825 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.2.1@KRBTEST.COM'} at 127.2.2.1:40245
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab.
Entry for principal kudu/127.2.2.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab.
I20260619 14:34:44.267324 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.2:0
--local_ip_for_outbound_sockets=127.2.2.2
--webserver_interface=127.2.2.2
--webserver_port=0
--tserver_master_addrs=127.2.2.62:46797
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab
--principal=kudu/127.2.2.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:39547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:34:44.379482 16021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:34:44.379763 16021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:34:44.379807 16021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:34:44.384016 16021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:34:44.384094 16021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:34:44.384156 16021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.2
I20260619 14:34:44.389870 16021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:39547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab
--principal=kudu/127.2.2.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.2.2
--webserver_port=0
--tserver_master_addrs=127.2.2.62:46797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.2
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:34:44.391069 16021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:34:44.392117 16021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:34:44.402187 16026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:34:44.402180 16029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:34:44.402182 16027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:34:44.402407 16021 server_base.cc:1034] running on GCE node
I20260619 14:34:44.403018 16021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:34:44.403681 16021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:34:44.404856 16021 hybrid_clock.cc:648] HybridClock initialized: now 1781879684404816 us; error 54 us; skew 500 ppm
Jun 19 14:34:44 dist-test-slave-70rm krb5kdc[15780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879684, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:34:44.407727 16021 init.cc:377] Logged in from keytab as kudu/127.2.2.2@KRBTEST.COM (short username kudu)
I20260619 14:34:44.408723 16021 webserver.cc:460] Webserver started at http://127.2.2.2:44979/ using document root <none> and password file <none>
I20260619 14:34:44.409334 16021 fs_manager.cc:359] Metadata directory not provided
I20260619 14:34:44.409431 16021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:34:44.409672 16021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:34:44.411837 16021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-1/data/instance:
uuid: "55a33368758b4a86a88156168ef4c6ec"
format_stamp: "Formatted at 2026-06-19 14:34:44 on dist-test-slave-70rm"
server_key: "809f69815f8eca585e8ea3aca958ee8b"
server_key_iv: "732a1081bc3a6ee3233fb5a1cdb2f226"
server_key_version: "encryptionkey@0"
I20260619 14:34:44.412427 16021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-1/wal/instance:
uuid: "55a33368758b4a86a88156168ef4c6ec"
format_stamp: "Formatted at 2026-06-19 14:34:44 on dist-test-slave-70rm"
server_key: "809f69815f8eca585e8ea3aca958ee8b"
server_key_iv: "732a1081bc3a6ee3233fb5a1cdb2f226"
server_key_version: "encryptionkey@0"
I20260619 14:34:44.416322 16021 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:34:44.419569 16036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:34:44.420811 16021 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 14:34:44.420957 16021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-1/wal
uuid: "55a33368758b4a86a88156168ef4c6ec"
format_stamp: "Formatted at 2026-06-19 14:34:44 on dist-test-slave-70rm"
server_key: "809f69815f8eca585e8ea3aca958ee8b"
server_key_iv: "732a1081bc3a6ee3233fb5a1cdb2f226"
server_key_version: "encryptionkey@0"
I20260619 14:34:44.421089 16021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:34:44.436054 16021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:34:44.439673 16021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:34:44.439894 16021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:34:44.440692 16021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:34:44.441597 16021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:34:44.441646 16021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:34:44.441689 16021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:34:44.441720 16021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:34:44.451010 16021 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.2:43741
I20260619 14:34:44.451020 16149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.2:43741 every 8 connection(s)
I20260619 14:34:44.451938 16021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
I20260619 14:34:44.453912 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 16021
I20260619 14:34:44.454037 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-1/wal/instance
I20260619 14:34:44.454350 2056 external_mini_cluster.cc:1428] Setting key aab543ab75a4e07274a489868372c4a1
Jun 19 14:34:44 dist-test-slave-70rm krb5kdc[15780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879684, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.2@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.2.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.3@KRBTEST.COM" created.
I20260619 14:34:44.467413 15864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:44.453775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:38385 (local address 127.2.2.62:46797)
0619 14:34:44.453908 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 14:34:44.453912 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:34:44.455364 (+ 1452us) server_negotiation.cc:407] Connection header received
0619 14:34:44.456420 (+ 1056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:34:44.456423 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:34:44.456473 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:34:44.456559 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:34:44.457977 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:44.458583 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:44.460426 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:44.460656 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:44.460761 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:44.463770 (+ 3009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:34:44.463774 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:34:44.463776 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:34:44.463800 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:34:44.465282 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:44.465793 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:44.465795 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:44.465796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:44.465867 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:44.466263 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:44.466265 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:44.466266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:44.466443 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:34:44.466521 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 14:34:44.467105 (+ 584us) server_negotiation.cc:299] Negotiation successful
0619 14:34:44.467238 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 14:34:44.468223 16152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:44.454352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46797 (local address 127.2.2.2:38385)
0619 14:34:44.455202 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:44.455260 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 14:34:44.456215 (+ 955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:34:44.456725 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:34:44.456732 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:34:44.457110 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:34:44.457814 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:44.457837 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:44.458714 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:44.458720 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:44.460283 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:44.460294 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:44.460777 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:44.460848 (+ 71us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:44.461103 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:44.461773 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:34:44.461802 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:34:44.463580 (+ 1778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:34:44.465426 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:34:44.465431 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:34:44.465445 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:34:44.465706 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:34:44.465945 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:34:44.465947 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:34:44.465948 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:34:44.466174 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:34:44.466537 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:34:44.466543 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:34:44.466825 (+ 282us) client_negotiation.cc:769] Sending connection context
0619 14:34:44.467026 (+ 201us) client_negotiation.cc:240] Negotiation successful
0619 14:34:44.467370 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":157,"threads_started":1}
I20260619 14:34:44.469530 16150 heartbeater.cc:347] Connected to a master server at 127.2.2.62:46797
I20260619 14:34:44.469800 16150 heartbeater.cc:464] Registering TS with master...
I20260619 14:34:44.470372 16150 heartbeater.cc:511] Master 127.2.2.62:46797 requested a full tablet report, sending...
I20260619 14:34:44.471487 15825 ts_manager.cc:194] Registered new tserver with Master: 55a33368758b4a86a88156168ef4c6ec (127.2.2.2:43741)
I20260619 14:34:44.472388 15825 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.2.2@KRBTEST.COM'} at 127.2.2.2:38385
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab.
Entry for principal kudu/127.2.2.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab.
I20260619 14:34:44.482936 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.3:0
--local_ip_for_outbound_sockets=127.2.2.3
--webserver_interface=127.2.2.3
--webserver_port=0
--tserver_master_addrs=127.2.2.62:46797
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab
--principal=kudu/127.2.2.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:39547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:34:44.596031 16155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:34:44.596311 16155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:34:44.596356 16155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:34:44.600607 16155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:34:44.600670 16155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:34:44.600729 16155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.3
I20260619 14:34:44.605937 16155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:39547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab
--principal=kudu/127.2.2.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.2.3
--webserver_port=0
--tserver_master_addrs=127.2.2.62:46797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.3
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:34:44.607002 16155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:34:44.607945 16155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:34:44.617694 16163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:34:44.617694 16160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:34:44.617805 16155 server_base.cc:1034] running on GCE node
W20260619 14:34:44.617695 16161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:34:44.618412 16155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:34:44.619127 16155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:34:44.620299 16155 hybrid_clock.cc:648] HybridClock initialized: now 1781879684620261 us; error 50 us; skew 500 ppm
Jun 19 14:34:44 dist-test-slave-70rm krb5kdc[15780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879684, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:34:44.623355 16155 init.cc:377] Logged in from keytab as kudu/127.2.2.3@KRBTEST.COM (short username kudu)
I20260619 14:34:44.624378 16155 webserver.cc:460] Webserver started at http://127.2.2.3:45009/ using document root <none> and password file <none>
I20260619 14:34:44.624969 16155 fs_manager.cc:359] Metadata directory not provided
I20260619 14:34:44.625062 16155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:34:44.625294 16155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:34:44.627516 16155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-2/data/instance:
uuid: "6c43506880b44e55b9fdfefbb44f2845"
format_stamp: "Formatted at 2026-06-19 14:34:44 on dist-test-slave-70rm"
server_key: "82a6f7b11a093d9d44720b057201e669"
server_key_iv: "fb581898a9f147bf76482f2eb9f83942"
server_key_version: "encryptionkey@0"
I20260619 14:34:44.628118 16155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-2/wal/instance:
uuid: "6c43506880b44e55b9fdfefbb44f2845"
format_stamp: "Formatted at 2026-06-19 14:34:44 on dist-test-slave-70rm"
server_key: "82a6f7b11a093d9d44720b057201e669"
server_key_iv: "fb581898a9f147bf76482f2eb9f83942"
server_key_version: "encryptionkey@0"
I20260619 14:34:44.632189 16155 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:34:44.634940 16170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:34:44.636361 16155 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:34:44.636523 16155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-2/wal
uuid: "6c43506880b44e55b9fdfefbb44f2845"
format_stamp: "Formatted at 2026-06-19 14:34:44 on dist-test-slave-70rm"
server_key: "82a6f7b11a093d9d44720b057201e669"
server_key_iv: "fb581898a9f147bf76482f2eb9f83942"
server_key_version: "encryptionkey@0"
I20260619 14:34:44.636656 16155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:34:44.648368 16155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:34:44.651553 16155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:34:44.651758 16155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:34:44.652361 16155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:34:44.653190 16155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:34:44.653259 16155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:34:44.653332 16155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:34:44.653383 16155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:34:44.662642 16155 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.3:34689
I20260619 14:34:44.662711 16283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.3:34689 every 8 connection(s)
I20260619 14:34:44.663626 16155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
I20260619 14:34:44.669836 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 16155
I20260619 14:34:44.669997 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-2/wal/instance
I20260619 14:34:44.670339 2056 external_mini_cluster.cc:1428] Setting key a88cdd9b302317b76e58212f582bcc43
Jun 19 14:34:44 dist-test-slave-70rm krb5kdc[15780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879684, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.3@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:34:44.678298 15864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:44.665755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.3:51613 (local address 127.2.2.62:46797)
0619 14:34:44.665929 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 14:34:44.665932 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:34:44.667093 (+ 1161us) server_negotiation.cc:407] Connection header received
0619 14:34:44.668053 (+ 960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:34:44.668057 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:34:44.668115 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:34:44.668229 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:34:44.669496 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:44.670177 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:44.671742 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:44.672030 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:44.672124 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:44.674692 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:34:44.674696 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:34:44.674698 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:34:44.674723 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:34:44.676268 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:44.676758 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:44.676761 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:44.676762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:44.676859 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:44.677208 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:44.677211 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:44.677212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:44.677425 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:34:44.677562 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 14:34:44.678007 (+ 445us) server_negotiation.cc:299] Negotiation successful
0619 14:34:44.678160 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 14:34:44.679266 16286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:44.666240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46797 (local address 127.2.2.3:51613)
0619 14:34:44.666939 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:44.666983 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 14:34:44.667833 (+ 850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:34:44.668423 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:34:44.668430 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:34:44.668841 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:34:44.669349 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:44.669361 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:44.670305 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:44.670308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:44.671570 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:44.671581 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:44.672135 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:44.672193 (+ 58us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:44.672395 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:44.673022 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:34:44.673038 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:34:44.674529 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:34:44.676369 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:34:44.676374 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:34:44.676388 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:34:44.676651 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:34:44.676945 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:34:44.676947 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:34:44.676948 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:34:44.677083 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:34:44.677526 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:34:44.677531 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:34:44.677772 (+ 241us) client_negotiation.cc:769] Sending connection context
0619 14:34:44.677959 (+ 187us) client_negotiation.cc:240] Negotiation successful
0619 14:34:44.678300 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":123,"threads_started":1}
I20260619 14:34:44.680498 16284 heartbeater.cc:347] Connected to a master server at 127.2.2.62:46797
I20260619 14:34:44.680749 16284 heartbeater.cc:464] Registering TS with master...
I20260619 14:34:44.681269 16284 heartbeater.cc:511] Master 127.2.2.62:46797 requested a full tablet report, sending...
I20260619 14:34:44.682348 15825 ts_manager.cc:194] Registered new tserver with Master: 6c43506880b44e55b9fdfefbb44f2845 (127.2.2.3:34689)
I20260619 14:34:44.682909 15825 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.2.3@KRBTEST.COM'} at 127.2.2.3:51613
I20260619 14:34:44.684435 2056 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:34:44.693849 15864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:44.686372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38864 (local address 127.2.2.62:46797)
0619 14:34:44.686531 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 14:34:44.686537 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:34:44.686762 (+ 225us) server_negotiation.cc:407] Connection header received
0619 14:34:44.686829 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:34:44.686832 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:34:44.686871 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:34:44.686943 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:34:44.687683 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:44.688207 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:44.689039 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:44.689258 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:44.689340 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:44.690735 (+ 1395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:34:44.690738 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:34:44.690740 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:34:44.690762 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:34:44.692122 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:44.692534 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:44.692537 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:44.692539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:44.692621 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:44.692892 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:44.692895 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:44.692897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:44.693144 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:34:44.693264 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 14:34:44.693501 (+ 237us) server_negotiation.cc:299] Negotiation successful
0619 14:34:44.693661 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 14:34:44.696715 15825 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38864:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:34:44.699790 15825 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:34:44.714480 16296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:44.708271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.1:46097 (local address 127.0.0.1:34870)
0619 14:34:44.709182 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:44.709236 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 14:34:44.709384 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:34:44.710028 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:34:44.710036 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:34:44.710061 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:34:44.710224 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:44.710231 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:44.711438 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:44.711443 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:44.712725 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:44.712735 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:44.713869 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:44.713873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:44.713996 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:44.714019 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 14:34:44.714106 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 14:34:44.714280 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":813,"mutex_wait_us":3,"spinlock_wait_cycles":14336,"thread_start_us":94,"threads_started":1}
I20260619 14:34:44.714488 16299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:44.709010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53510 (local address 127.2.2.3:34689)
0619 14:34:44.709661 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0619 14:34:44.709677 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 14:34:44.710001 (+ 324us) server_negotiation.cc:407] Connection header received
0619 14:34:44.710079 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:34:44.710083 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:34:44.710297 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:34:44.710455 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:34:44.710648 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:44.711726 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:44.713079 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:44.713639 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:44.713763 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:44.713932 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 14:34:44.714221 (+ 289us) server_negotiation.cc:299] Negotiation successful
0619 14:34:44.714327 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":141,"threads_started":1}
I20260619 14:34:44.714529 16297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:44.708382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55244 (local address 127.2.2.2:43741)
0619 14:34:44.708681 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0619 14:34:44.708696 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:34:44.709527 (+ 831us) server_negotiation.cc:407] Connection header received
0619 14:34:44.709612 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:34:44.709617 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:34:44.709829 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:34:44.709984 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:34:44.710391 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:44.711604 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:44.713094 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:44.713880 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:44.713983 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:44.714158 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 14:34:44.714280 (+ 122us) server_negotiation.cc:299] Negotiation successful
0619 14:34:44.714387 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":103,"threads_started":1}
I20260619 14:34:44.714891 16298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:44.708830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.3:34689 (local address 127.0.0.1:53510)
0619 14:34:44.709808 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:44.709823 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:34:44.709907 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:34:44.710435 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:34:44.710439 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:34:44.710453 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:34:44.710547 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:44.710553 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:44.711855 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:44.711858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:44.712965 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:44.712972 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:44.713762 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:44.713765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:44.713953 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:44.714010 (+ 57us) client_negotiation.cc:769] Sending connection context
0619 14:34:44.714652 (+ 642us) client_negotiation.cc:240] Negotiation successful
0619 14:34:44.714754 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":917,"mutex_wait_us":53,"thread_start_us":91,"threads_started":1}
I20260619 14:34:44.714902 16300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:44.709087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34870 (local address 127.2.2.1:46097)
0619 14:34:44.709632 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0619 14:34:44.709647 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:34:44.709665 (+ 18us) server_negotiation.cc:407] Connection header received
0619 14:34:44.709726 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:34:44.709730 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:34:44.709866 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:34:44.710379 (+ 513us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:34:44.710558 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:44.711308 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:44.712859 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:44.713712 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:44.714420 (+ 708us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:44.714571 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 14:34:44.714694 (+ 123us) server_negotiation.cc:299] Negotiation successful
0619 14:34:44.714792 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":114,"threads_started":1}
I20260619 14:34:44.715209 16295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:44.708236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.2:43741 (local address 127.0.0.1:55244)
0619 14:34:44.709182 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:44.709238 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 14:34:44.709378 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:34:44.710184 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:34:44.710187 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:34:44.710201 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:34:44.710272 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:44.710280 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:44.711757 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:44.711760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:44.712948 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:44.712958 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:44.714007 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:44.714011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:44.714142 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:44.714173 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 14:34:44.714250 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 14:34:44.714324 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":754,"mutex_wait_us":37,"spinlock_wait_cycles":3968,"thread_start_us":85,"threads_started":1}
I20260619 14:34:44.717885 16084 tablet_service.cc:1467] Processing CreateTablet for tablet 388c90dd7e9e4ffc8d551ffc274d0ff7 (DEFAULT_TABLE table=test-table [id=addc849ecca84013828f726f5c546448]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:34:44.717943 16218 tablet_service.cc:1467] Processing CreateTablet for tablet 388c90dd7e9e4ffc8d551ffc274d0ff7 (DEFAULT_TABLE table=test-table [id=addc849ecca84013828f726f5c546448]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:34:44.717942 15950 tablet_service.cc:1467] Processing CreateTablet for tablet 388c90dd7e9e4ffc8d551ffc274d0ff7 (DEFAULT_TABLE table=test-table [id=addc849ecca84013828f726f5c546448]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:34:44.718954 16084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 388c90dd7e9e4ffc8d551ffc274d0ff7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:34:44.718978 16218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 388c90dd7e9e4ffc8d551ffc274d0ff7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:34:44.718974 15950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 388c90dd7e9e4ffc8d551ffc274d0ff7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:34:44.727155 16302 tablet_bootstrap.cc:492] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P c71ce75c24df481097464fb1957b76b4: Bootstrap starting.
I20260619 14:34:44.727679 16301 tablet_bootstrap.cc:492] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P 55a33368758b4a86a88156168ef4c6ec: Bootstrap starting.
I20260619 14:34:44.728003 16303 tablet_bootstrap.cc:492] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P 6c43506880b44e55b9fdfefbb44f2845: Bootstrap starting.
I20260619 14:34:44.729527 16301 tablet_bootstrap.cc:654] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P 55a33368758b4a86a88156168ef4c6ec: Neither blocks nor log segments found. Creating new log.
I20260619 14:34:44.729563 16302 tablet_bootstrap.cc:654] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P c71ce75c24df481097464fb1957b76b4: Neither blocks nor log segments found. Creating new log.
I20260619 14:34:44.729985 16303 tablet_bootstrap.cc:654] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P 6c43506880b44e55b9fdfefbb44f2845: Neither blocks nor log segments found. Creating new log.
I20260619 14:34:44.730842 16301 log.cc:824] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P 55a33368758b4a86a88156168ef4c6ec: Log is configured to *not* fsync() on all Append() calls
I20260619 14:34:44.730882 16302 log.cc:824] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P c71ce75c24df481097464fb1957b76b4: Log is configured to *not* fsync() on all Append() calls
I20260619 14:34:44.731283 16303 log.cc:824] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P 6c43506880b44e55b9fdfefbb44f2845: Log is configured to *not* fsync() on all Append() calls
I20260619 14:34:44.732901 16301 tablet_bootstrap.cc:492] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P 55a33368758b4a86a88156168ef4c6ec: No bootstrap required, opened a new log
I20260619 14:34:44.733119 16301 ts_tablet_manager.cc:1397] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P 55a33368758b4a86a88156168ef4c6ec: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 14:34:44.733330 16303 tablet_bootstrap.cc:492] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P 6c43506880b44e55b9fdfefbb44f2845: No bootstrap required, opened a new log
I20260619 14:34:44.733529 16303 ts_tablet_manager.cc:1397] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P 6c43506880b44e55b9fdfefbb44f2845: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 14:34:44.733781 16302 tablet_bootstrap.cc:492] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P c71ce75c24df481097464fb1957b76b4: No bootstrap required, opened a new log
I20260619 14:34:44.733951 16302 ts_tablet_manager.cc:1397] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P c71ce75c24df481097464fb1957b76b4: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260619 14:34:44.736385 16301 raft_consensus.cc:348] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P 55a33368758b4a86a88156168ef4c6ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a33368758b4a86a88156168ef4c6ec" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 43741 } } peers { permanent_uuid: "c71ce75c24df481097464fb1957b76b4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 46097 } } peers { permanent_uuid: "6c43506880b44e55b9fdfefbb44f2845" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 34689 } }
I20260619 14:34:44.736666 16301 raft_consensus.cc:374] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P 55a33368758b4a86a88156168ef4c6ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:34:44.736768 16301 raft_consensus.cc:729] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P 55a33368758b4a86a88156168ef4c6ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55a33368758b4a86a88156168ef4c6ec, State: Initialized, Role: FOLLOWER
I20260619 14:34:44.737257 16301 consensus_queue.cc:260] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P 55a33368758b4a86a88156168ef4c6ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a33368758b4a86a88156168ef4c6ec" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 43741 } } peers { permanent_uuid: "c71ce75c24df481097464fb1957b76b4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 46097 } } peers { permanent_uuid: "6c43506880b44e55b9fdfefbb44f2845" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 34689 } }
I20260619 14:34:44.737308 16303 raft_consensus.cc:348] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P 6c43506880b44e55b9fdfefbb44f2845 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a33368758b4a86a88156168ef4c6ec" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 43741 } } peers { permanent_uuid: "c71ce75c24df481097464fb1957b76b4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 46097 } } peers { permanent_uuid: "6c43506880b44e55b9fdfefbb44f2845" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 34689 } }
I20260619 14:34:44.737641 16303 raft_consensus.cc:374] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P 6c43506880b44e55b9fdfefbb44f2845 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:34:44.737736 16303 raft_consensus.cc:729] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P 6c43506880b44e55b9fdfefbb44f2845 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c43506880b44e55b9fdfefbb44f2845, State: Initialized, Role: FOLLOWER
I20260619 14:34:44.737684 16302 raft_consensus.cc:348] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P c71ce75c24df481097464fb1957b76b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a33368758b4a86a88156168ef4c6ec" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 43741 } } peers { permanent_uuid: "c71ce75c24df481097464fb1957b76b4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 46097 } } peers { permanent_uuid: "6c43506880b44e55b9fdfefbb44f2845" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 34689 } }
I20260619 14:34:44.737932 16302 raft_consensus.cc:374] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P c71ce75c24df481097464fb1957b76b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:34:44.737987 16302 raft_consensus.cc:729] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P c71ce75c24df481097464fb1957b76b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c71ce75c24df481097464fb1957b76b4, State: Initialized, Role: FOLLOWER
I20260619 14:34:44.738415 16150 heartbeater.cc:503] Master 127.2.2.62:46797 was elected leader, sending a full tablet report...
I20260619 14:34:44.738430 16301 ts_tablet_manager.cc:1428] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P 55a33368758b4a86a88156168ef4c6ec: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 14:34:44.738399 16303 consensus_queue.cc:260] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P 6c43506880b44e55b9fdfefbb44f2845 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a33368758b4a86a88156168ef4c6ec" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 43741 } } peers { permanent_uuid: "c71ce75c24df481097464fb1957b76b4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 46097 } } peers { permanent_uuid: "6c43506880b44e55b9fdfefbb44f2845" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 34689 } }
I20260619 14:34:44.738963 16302 consensus_queue.cc:260] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P c71ce75c24df481097464fb1957b76b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a33368758b4a86a88156168ef4c6ec" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 43741 } } peers { permanent_uuid: "c71ce75c24df481097464fb1957b76b4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 46097 } } peers { permanent_uuid: "6c43506880b44e55b9fdfefbb44f2845" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 34689 } }
I20260619 14:34:44.739645 16303 ts_tablet_manager.cc:1428] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P 6c43506880b44e55b9fdfefbb44f2845: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 14:34:44.740010 16284 heartbeater.cc:503] Master 127.2.2.62:46797 was elected leader, sending a full tablet report...
I20260619 14:34:44.740072 16302 ts_tablet_manager.cc:1428] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P c71ce75c24df481097464fb1957b76b4: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260619 14:34:44.740442 16016 heartbeater.cc:503] Master 127.2.2.62:46797 was elected leader, sending a full tablet report...
W20260619 14:34:44.914952 16285 tablet.cc:2369] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P 6c43506880b44e55b9fdfefbb44f2845: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:34:44.953115 16151 tablet.cc:2369] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P 55a33368758b4a86a88156168ef4c6ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:34:44.975052 16307 raft_consensus.cc:482] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P 55a33368758b4a86a88156168ef4c6ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:34:44.975250 16307 raft_consensus.cc:504] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P 55a33368758b4a86a88156168ef4c6ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a33368758b4a86a88156168ef4c6ec" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 43741 } } peers { permanent_uuid: "c71ce75c24df481097464fb1957b76b4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 46097 } } peers { permanent_uuid: "6c43506880b44e55b9fdfefbb44f2845" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 34689 } }
I20260619 14:34:44.976433 16307 leader_election.cc:288] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P 55a33368758b4a86a88156168ef4c6ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c71ce75c24df481097464fb1957b76b4 (127.2.2.1:46097), 6c43506880b44e55b9fdfefbb44f2845 (127.2.2.3:34689)
I20260619 14:34:44.980705 16299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:44.976799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:39965 (local address 127.2.2.3:34689)
0619 14:34:44.977020 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 14:34:44.977027 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:34:44.977191 (+ 164us) server_negotiation.cc:407] Connection header received
0619 14:34:44.977326 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:34:44.977329 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:34:44.977371 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:34:44.977422 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:34:44.977770 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:44.978413 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:44.979649 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:44.980106 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:44.980184 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:44.980263 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 14:34:44.980453 (+ 190us) server_negotiation.cc:299] Negotiation successful
0619 14:34:44.980560 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260619 14:34:44.980748 16300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:44.976754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:56011 (local address 127.2.2.1:46097)
0619 14:34:44.976903 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 14:34:44.976908 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:34:44.977111 (+ 203us) server_negotiation.cc:407] Connection header received
0619 14:34:44.977295 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:34:44.977301 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:34:44.977355 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:34:44.977472 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:34:44.977841 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:44.978480 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:44.979701 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:44.980178 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:44.980283 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:44.980342 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 14:34:44.980542 (+ 200us) server_negotiation.cc:299] Negotiation successful
0619 14:34:44.980624 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 14:34:44.980697 16310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:44.976685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.3:34689 (local address 127.2.2.2:39965)
0619 14:34:44.977064 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:44.977097 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 14:34:44.977208 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:34:44.977461 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:34:44.977465 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:34:44.977481 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:34:44.977654 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:44.977660 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:44.978517 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:44.978524 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:44.979527 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:44.979539 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:44.980219 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:44.980224 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:44.980325 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:44.980350 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 14:34:44.980425 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 14:34:44.980517 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":5,"spinlock_wait_cycles":23296,"thread_start_us":110,"threads_started":1}
I20260619 14:34:44.980726 16311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:44.976662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.1:46097 (local address 127.2.2.2:56011)
0619 14:34:44.976982 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:44.976997 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:34:44.977142 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:34:44.977496 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:34:44.977499 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:34:44.977540 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:34:44.977729 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:44.977738 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:44.978574 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:44.978577 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:44.979555 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:44.979572 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:44.980300 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:44.980303 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:44.980390 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:44.980433 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 14:34:44.980507 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 14:34:44.980581 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":39,"spinlock_wait_cycles":768,"thread_start_us":74,"threads_started":1}
I20260619 14:34:44.981479 16238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "388c90dd7e9e4ffc8d551ffc274d0ff7" candidate_uuid: "55a33368758b4a86a88156168ef4c6ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c43506880b44e55b9fdfefbb44f2845" is_pre_election: true
I20260619 14:34:44.981572 15970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "388c90dd7e9e4ffc8d551ffc274d0ff7" candidate_uuid: "55a33368758b4a86a88156168ef4c6ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c71ce75c24df481097464fb1957b76b4" is_pre_election: true
I20260619 14:34:44.981827 16238 raft_consensus.cc:2454] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P 6c43506880b44e55b9fdfefbb44f2845 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55a33368758b4a86a88156168ef4c6ec in term 0.
I20260619 14:34:44.981873 15970 raft_consensus.cc:2454] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P c71ce75c24df481097464fb1957b76b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55a33368758b4a86a88156168ef4c6ec in term 0.
I20260619 14:34:44.982327 16040 leader_election.cc:302] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P 55a33368758b4a86a88156168ef4c6ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55a33368758b4a86a88156168ef4c6ec, 6c43506880b44e55b9fdfefbb44f2845; no voters:
I20260619 14:34:44.982622 16307 raft_consensus.cc:2781] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P 55a33368758b4a86a88156168ef4c6ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:34:44.982707 16307 raft_consensus.cc:482] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P 55a33368758b4a86a88156168ef4c6ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:34:44.982750 16307 raft_consensus.cc:3037] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P 55a33368758b4a86a88156168ef4c6ec [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:34:44.983795 16307 raft_consensus.cc:504] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P 55a33368758b4a86a88156168ef4c6ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a33368758b4a86a88156168ef4c6ec" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 43741 } } peers { permanent_uuid: "c71ce75c24df481097464fb1957b76b4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 46097 } } peers { permanent_uuid: "6c43506880b44e55b9fdfefbb44f2845" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 34689 } }
I20260619 14:34:44.984138 16307 leader_election.cc:288] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P 55a33368758b4a86a88156168ef4c6ec [CANDIDATE]: Term 1 election: Requested vote from peers c71ce75c24df481097464fb1957b76b4 (127.2.2.1:46097), 6c43506880b44e55b9fdfefbb44f2845 (127.2.2.3:34689)
I20260619 14:34:44.984432 15970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "388c90dd7e9e4ffc8d551ffc274d0ff7" candidate_uuid: "55a33368758b4a86a88156168ef4c6ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c71ce75c24df481097464fb1957b76b4"
I20260619 14:34:44.984469 16238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "388c90dd7e9e4ffc8d551ffc274d0ff7" candidate_uuid: "55a33368758b4a86a88156168ef4c6ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c43506880b44e55b9fdfefbb44f2845"
I20260619 14:34:44.984598 15970 raft_consensus.cc:3037] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P c71ce75c24df481097464fb1957b76b4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:34:44.984637 16238 raft_consensus.cc:3037] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P 6c43506880b44e55b9fdfefbb44f2845 [term 0 FOLLOWER]: Advancing to term 1
W20260619 14:34:44.985589 16017 tablet.cc:2369] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P c71ce75c24df481097464fb1957b76b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:34:44.985765 15970 raft_consensus.cc:2454] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P c71ce75c24df481097464fb1957b76b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55a33368758b4a86a88156168ef4c6ec in term 1.
I20260619 14:34:44.985786 16238 raft_consensus.cc:2454] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P 6c43506880b44e55b9fdfefbb44f2845 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55a33368758b4a86a88156168ef4c6ec in term 1.
I20260619 14:34:44.986053 16040 leader_election.cc:302] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P 55a33368758b4a86a88156168ef4c6ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55a33368758b4a86a88156168ef4c6ec, 6c43506880b44e55b9fdfefbb44f2845; no voters:
I20260619 14:34:44.986261 16307 raft_consensus.cc:2781] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P 55a33368758b4a86a88156168ef4c6ec [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:34:44.986478 16307 raft_consensus.cc:686] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P 55a33368758b4a86a88156168ef4c6ec [term 1 LEADER]: Becoming Leader. State: Replica: 55a33368758b4a86a88156168ef4c6ec, State: Running, Role: LEADER
I20260619 14:34:44.986814 16307 consensus_queue.cc:237] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P 55a33368758b4a86a88156168ef4c6ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a33368758b4a86a88156168ef4c6ec" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 43741 } } peers { permanent_uuid: "c71ce75c24df481097464fb1957b76b4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 46097 } } peers { permanent_uuid: "6c43506880b44e55b9fdfefbb44f2845" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 34689 } }
I20260619 14:34:44.990338 15824 catalog_manager.cc:5495] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P 55a33368758b4a86a88156168ef4c6ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 55a33368758b4a86a88156168ef4c6ec (127.2.2.2). New cstate: current_term: 1 leader_uuid: "55a33368758b4a86a88156168ef4c6ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a33368758b4a86a88156168ef4c6ec" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 43741 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c71ce75c24df481097464fb1957b76b4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 46097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c43506880b44e55b9fdfefbb44f2845" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 34689 } health_report { overall_health: UNKNOWN } } }
I20260619 14:34:45.007351 16297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:45.003781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55248 (local address 127.2.2.2:43741)
0619 14:34:45.003933 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 14:34:45.003938 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:34:45.003952 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:34:45.004129 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:34:45.004132 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:34:45.004189 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:34:45.004299 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:34:45.004542 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:45.005042 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:45.005881 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:45.006101 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:45.006260 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:45.006560 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:34:45.006978 (+ 418us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:34:45.007113 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 14:34:45.007149 (+ 36us) server_negotiation.cc:299] Negotiation successful
0619 14:34:45.007203 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 14:34:45.013216 16238 raft_consensus.cc:1261] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P 6c43506880b44e55b9fdfefbb44f2845 [term 1 FOLLOWER]: Refusing update from remote peer 55a33368758b4a86a88156168ef4c6ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:34:45.013216 15970 raft_consensus.cc:1261] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P c71ce75c24df481097464fb1957b76b4 [term 1 FOLLOWER]: Refusing update from remote peer 55a33368758b4a86a88156168ef4c6ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:34:45.013963 16312 consensus_queue.cc:1035] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P 55a33368758b4a86a88156168ef4c6ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c43506880b44e55b9fdfefbb44f2845" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 34689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:34:45.014166 16307 consensus_queue.cc:1035] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P 55a33368758b4a86a88156168ef4c6ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "c71ce75c24df481097464fb1957b76b4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:34:45.022634 16316 mvcc.cc:205] Tried to move back new op lower bound from 7298579189807038464 to 7298579189708787712. Current Snapshot: MvccSnapshot[applied={T|T < 7298579189807038464}]
I20260619 14:34:45.030682 15824 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:34:45.030885 15824 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:34:45.033316 15824 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986: Sending DeleteTablet for 3 replicas of tablet 388c90dd7e9e4ffc8d551ffc274d0ff7
I20260619 14:34:45.034183 16084 tablet_service.cc:1514] Processing DeleteTablet for tablet 388c90dd7e9e4ffc8d551ffc274d0ff7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:34:45 UTC) from {username='kudu', principal='kudu/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:55244
I20260619 14:34:45.034261 15950 tablet_service.cc:1514] Processing DeleteTablet for tablet 388c90dd7e9e4ffc8d551ffc274d0ff7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:34:45 UTC) from {username='kudu', principal='kudu/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:34870
I20260619 14:34:45.034436 16218 tablet_service.cc:1514] Processing DeleteTablet for tablet 388c90dd7e9e4ffc8d551ffc274d0ff7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:34:45 UTC) from {username='kudu', principal='kudu/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:53510
I20260619 14:34:45.034795 16326 tablet_replica.cc:330] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P c71ce75c24df481097464fb1957b76b4: stopping tablet replica
I20260619 14:34:45.034900 16325 tablet_replica.cc:330] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P 55a33368758b4a86a88156168ef4c6ec: stopping tablet replica
I20260619 14:34:45.034986 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 15887
I20260619 14:34:45.035209 16325 raft_consensus.cc:2229] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P 55a33368758b4a86a88156168ef4c6ec [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:34:45.035580 16325 raft_consensus.cc:2258] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P 55a33368758b4a86a88156168ef4c6ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:34:45.035912 16327 tablet_replica.cc:330] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P 6c43506880b44e55b9fdfefbb44f2845: stopping tablet replica
I20260619 14:34:45.036175 16327 raft_consensus.cc:2229] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P 6c43506880b44e55b9fdfefbb44f2845 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:34:45.036463 16327 raft_consensus.cc:2258] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P 6c43506880b44e55b9fdfefbb44f2845 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:34:45.038690 16327 ts_tablet_manager.cc:1905] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P 6c43506880b44e55b9fdfefbb44f2845: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:34:45.038690 16325 ts_tablet_manager.cc:1905] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P 55a33368758b4a86a88156168ef4c6ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:34:45.041877 16327 ts_tablet_manager.cc:1918] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P 6c43506880b44e55b9fdfefbb44f2845: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:34:45.041864 16325 ts_tablet_manager.cc:1918] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P 55a33368758b4a86a88156168ef4c6ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:34:45.041993 16325 log.cc:1196] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P 55a33368758b4a86a88156168ef4c6ec: Deleting WAL directory at /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-1/wal/wals/388c90dd7e9e4ffc8d551ffc274d0ff7
I20260619 14:34:45.042071 16327 log.cc:1196] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P 6c43506880b44e55b9fdfefbb44f2845: Deleting WAL directory at /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-2/wal/wals/388c90dd7e9e4ffc8d551ffc274d0ff7
I20260619 14:34:45.042505 16325 ts_tablet_manager.cc:1939] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P 55a33368758b4a86a88156168ef4c6ec: Deleting consensus metadata
W20260619 14:34:45.042621 15810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.2.1:46097 (error 108)
I20260619 14:34:45.043205 16327 ts_tablet_manager.cc:1939] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P 6c43506880b44e55b9fdfefbb44f2845: Deleting consensus metadata
I20260619 14:34:45.043469 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 16021
I20260619 14:34:45.044837 16298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:45.043775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.1:46097 (local address 127.0.0.1:34878)
0619 14:34:45.044677 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:45.044748 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.1:46097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":819}
W20260619 14:34:45.045019 15810 catalog_manager.cc:4584] TS c71ce75c24df481097464fb1957b76b4 (127.2.2.1:46097): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 388c90dd7e9e4ffc8d551ffc274d0ff7: Network error: Client connection negotiation failed: client connection to 127.2.2.1:46097: connect: Connection refused (error 111)
I20260619 14:34:45.046406 15812 catalog_manager.cc:4841] TS 6c43506880b44e55b9fdfefbb44f2845 (127.2.2.3:34689): tablet 388c90dd7e9e4ffc8d551ffc274d0ff7 (table test-table [id=addc849ecca84013828f726f5c546448]) successfully deleted
I20260619 14:34:45.049865 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 16155
I20260619 14:34:45.050769 16298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:45.050068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.2:43741 (local address 127.0.0.1:55250)
0619 14:34:45.050593 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:45.050661 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.2:43741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421}
I20260619 14:34:45.056691 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 15793
I20260619 14:34:45.063074 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:46797
--webserver_interface=127.2.2.62
--webserver_port=45891
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:39547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.2.62:46797
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:34:45.174017 16329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:34:45.174378 16329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:34:45.174459 16329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:34:45.178757 16329 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:34:45.178864 16329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:34:45.178903 16329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:34:45.178957 16329 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:34:45.178993 16329 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:34:45.184502 16329 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:39547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.2.2.62:46797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:46797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.2.62
--webserver_port=45891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:34:45.185745 16329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:34:45.186878 16329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:34:45.200429 16335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:34:45.200429 16336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:34:45.200450 16338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:34:45.200835 16329 server_base.cc:1034] running on GCE node
I20260619 14:34:45.201483 16329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:34:45.202788 16329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:34:45.203977 16329 hybrid_clock.cc:648] HybridClock initialized: now 1781879685203962 us; error 36 us; skew 500 ppm
Jun 19 14:34:45 dist-test-slave-70rm krb5kdc[15780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879685, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:34:45.206924 16329 init.cc:377] Logged in from keytab as kudu/127.2.2.62@KRBTEST.COM (short username kudu)
I20260619 14:34:45.207926 16329 webserver.cc:460] Webserver started at http://127.2.2.62:45891/ using document root <none> and password file <none>
I20260619 14:34:45.208530 16329 fs_manager.cc:359] Metadata directory not provided
I20260619 14:34:45.208643 16329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:34:45.212642 16329 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 14:34:45.214915 16345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:34:45.216185 16329 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:34:45.216333 16329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/master-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/master-0/wal
uuid: "f971ee13217e41268e22d92d2a111986"
format_stamp: "Formatted at 2026-06-19 14:34:43 on dist-test-slave-70rm"
server_key: "4e56bd85d38ea569d4be8f50d6aaa9ad"
server_key_iv: "d022057fa1f55ec60b11a6bb14e89137"
server_key_version: "encryptionkey@0"
I20260619 14:34:45.216965 16329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:34:45.236495 16329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:34:45.239548 16329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:34:45.239763 16329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:34:45.247371 16329 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.62:46797
I20260619 14:34:45.247388 16397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.62:46797 every 8 connection(s)
I20260619 14:34:45.248505 16329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
I20260619 14:34:45.249797 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 16329
I20260619 14:34:45.250329 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.1:46097
--local_ip_for_outbound_sockets=127.2.2.1
--tserver_master_addrs=127.2.2.62:46797
--webserver_port=38943
--webserver_interface=127.2.2.1
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab
--principal=kudu/127.2.2.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:39547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:34:45.253870 16398 sys_catalog.cc:263] Verifying existing consensus state
I20260619 14:34:45.255643 16398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986: Bootstrap starting.
I20260619 14:34:45.272691 16398 log.cc:824] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986: Log is configured to *not* fsync() on all Append() calls
I20260619 14:34:45.281586 16398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:34:45.282030 16398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986: Bootstrap complete.
I20260619 14:34:45.286679 16398 raft_consensus.cc:348] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f971ee13217e41268e22d92d2a111986" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 46797 } }
I20260619 14:34:45.287110 16398 raft_consensus.cc:729] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f971ee13217e41268e22d92d2a111986, State: Initialized, Role: FOLLOWER
I20260619 14:34:45.287611 16398 consensus_queue.cc:260] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f971ee13217e41268e22d92d2a111986" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 46797 } }
I20260619 14:34:45.287724 16398 raft_consensus.cc:388] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:34:45.287819 16398 raft_consensus.cc:482] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:34:45.287943 16398 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986 [term 1 FOLLOWER]: Advancing to term 2
I20260619 14:34:45.290196 16398 raft_consensus.cc:504] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f971ee13217e41268e22d92d2a111986" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 46797 } }
I20260619 14:34:45.290556 16398 leader_election.cc:302] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f971ee13217e41268e22d92d2a111986; no voters:
I20260619 14:34:45.290953 16398 leader_election.cc:288] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 14:34:45.291024 16402 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 14:34:45.291276 16402 raft_consensus.cc:686] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986 [term 2 LEADER]: Becoming Leader. State: Replica: f971ee13217e41268e22d92d2a111986, State: Running, Role: LEADER
I20260619 14:34:45.291647 16402 consensus_queue.cc:237] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f971ee13217e41268e22d92d2a111986" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 46797 } }
I20260619 14:34:45.292268 16398 sys_catalog.cc:564] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:34:45.292662 16402 sys_catalog.cc:455] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f971ee13217e41268e22d92d2a111986. Latest consensus state: current_term: 2 leader_uuid: "f971ee13217e41268e22d92d2a111986" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f971ee13217e41268e22d92d2a111986" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 46797 } } }
I20260619 14:34:45.292816 16402 sys_catalog.cc:458] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986 [sys.catalog]: This master's current role is: LEADER
I20260619 14:34:45.293227 16406 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:34:45.293246 16403 sys_catalog.cc:455] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f971ee13217e41268e22d92d2a111986" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f971ee13217e41268e22d92d2a111986" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 46797 } } }
I20260619 14:34:45.293540 16403 sys_catalog.cc:458] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986 [sys.catalog]: This master's current role is: LEADER
I20260619 14:34:45.298403 16406 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:34:45.299146 16406 catalog_manager.cc:1259] Loaded cluster ID: f623e242ae6241d69aa547d8601a075c
I20260619 14:34:45.299234 16406 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:34:45.300343 16406 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260619 14:34:45.303148 16406 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260619 14:34:45.308145 16406 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:34:45.308933 16406 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986: Loaded TSK: 0
I20260619 14:34:45.309991 16406 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 14:34:45.376976 16400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:34:45.377287 16400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:34:45.377360 16400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:34:45.381440 16400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:34:45.381541 16400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:34:45.381637 16400 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.1
I20260619 14:34:45.387518 16400 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:39547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab
--principal=kudu/127.2.2.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.1:46097
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.2.1
--webserver_port=38943
--tserver_master_addrs=127.2.2.62:46797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.1
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:34:45.388690 16400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:34:45.389875 16400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:34:45.400966 16425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:34:45.400978 16428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:34:45.401132 16426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:34:45.401355 16400 server_base.cc:1034] running on GCE node
I20260619 14:34:45.401795 16400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:34:45.402611 16400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:34:45.403839 16400 hybrid_clock.cc:648] HybridClock initialized: now 1781879685403802 us; error 47 us; skew 500 ppm
Jun 19 14:34:45 dist-test-slave-70rm krb5kdc[15780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879685, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:34:45.406870 16400 init.cc:377] Logged in from keytab as kudu/127.2.2.1@KRBTEST.COM (short username kudu)
I20260619 14:34:45.407848 16400 webserver.cc:460] Webserver started at http://127.2.2.1:38943/ using document root <none> and password file <none>
I20260619 14:34:45.408480 16400 fs_manager.cc:359] Metadata directory not provided
I20260619 14:34:45.408557 16400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:34:45.412676 16400 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:34:45.415026 16435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:34:45.416414 16400 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:34:45.416639 16400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-0/wal
uuid: "c71ce75c24df481097464fb1957b76b4"
format_stamp: "Formatted at 2026-06-19 14:34:44 on dist-test-slave-70rm"
server_key: "a51a61d3af2563badf3e5fbec931e7cc"
server_key_iv: "cdb1c3a55a66061db5e6c614947ba594"
server_key_version: "encryptionkey@0"
I20260619 14:34:45.417197 16400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:34:45.441210 16400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:34:45.444751 16400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:34:45.444965 16400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:34:45.445653 16400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:34:45.446874 16442 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 14:34:45.451128 16400 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 14:34:45.451195 16400 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260619 14:34:45.451244 16400 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 14:34:45.452883 16400 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 14:34:45.452942 16400 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260619 14:34:45.453208 16442 tablet_bootstrap.cc:492] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P c71ce75c24df481097464fb1957b76b4: Bootstrap starting.
I20260619 14:34:45.463945 16400 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.1:46097
I20260619 14:34:45.463963 16549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.1:46097 every 8 connection(s)
I20260619 14:34:45.465178 16400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
I20260619 14:34:45.467515 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 16400
I20260619 14:34:45.467993 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.2:43741
--local_ip_for_outbound_sockets=127.2.2.2
--tserver_master_addrs=127.2.2.62:46797
--webserver_port=44979
--webserver_interface=127.2.2.2
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab
--principal=kudu/127.2.2.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:39547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:34:45.474010 16442 log.cc:824] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P c71ce75c24df481097464fb1957b76b4: Log is configured to *not* fsync() on all Append() calls
Jun 19 14:34:45 dist-test-slave-70rm krb5kdc[15780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879685, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.1@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:34:45.483229 16442 tablet_bootstrap.cc:492] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P c71ce75c24df481097464fb1957b76b4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:34:45.483703 16442 tablet_bootstrap.cc:492] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P c71ce75c24df481097464fb1957b76b4: Bootstrap complete.
I20260619 14:34:45.483764 16553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:45.467291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46797 (local address 127.2.2.1:35149)
0619 14:34:45.467934 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:45.467993 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 14:34:45.469282 (+ 1289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:34:45.470508 (+ 1226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:34:45.470517 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:34:45.470952 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:34:45.471411 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:45.471424 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:45.472738 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:45.472741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:45.473584 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:45.473593 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:45.474365 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:45.474404 (+ 39us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:45.474597 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:45.475253 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:34:45.475273 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:34:45.477395 (+ 2122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:34:45.480049 (+ 2654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:34:45.480057 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:34:45.480073 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:34:45.480473 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:34:45.480901 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:34:45.480906 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:34:45.480911 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:34:45.481103 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:34:45.481818 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:34:45.481826 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:34:45.482206 (+ 380us) client_negotiation.cc:769] Sending connection context
0619 14:34:45.482449 (+ 243us) client_negotiation.cc:240] Negotiation successful
0619 14:34:45.482821 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":161,"threads_started":1}
I20260619 14:34:45.483757 16552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:45.467299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:35149 (local address 127.2.2.62:46797)
0619 14:34:45.468534 (+ 1235us) server_negotiation.cc:206] Beginning negotiation
0619 14:34:45.468552 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 14:34:45.468599 (+ 47us) server_negotiation.cc:407] Connection header received
0619 14:34:45.469754 (+ 1155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:34:45.469765 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:34:45.470059 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:34:45.470454 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:34:45.471596 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:45.472570 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:45.473710 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:45.474259 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:45.474435 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:45.477671 (+ 3236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:34:45.477678 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:34:45.477694 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:34:45.477730 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:34:45.479906 (+ 2176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:45.480609 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:45.480615 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:45.480620 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:45.480776 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:45.481218 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:45.481221 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:45.481222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:45.481673 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:34:45.481791 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 14:34:45.482572 (+ 781us) server_negotiation.cc:299] Negotiation successful
0619 14:34:45.482868 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1073,"thread_start_us":113,"threads_started":1}
I20260619 14:34:45.485057 16442 ts_tablet_manager.cc:1397] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P c71ce75c24df481097464fb1957b76b4: Time spent bootstrapping tablet: real 0.032s user 0.023s sys 0.008s
I20260619 14:34:45.486665 16550 heartbeater.cc:347] Connected to a master server at 127.2.2.62:46797
I20260619 14:34:45.486901 16550 heartbeater.cc:464] Registering TS with master...
I20260619 14:34:45.487423 16550 heartbeater.cc:511] Master 127.2.2.62:46797 requested a full tablet report, sending...
I20260619 14:34:45.487896 16442 raft_consensus.cc:348] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P c71ce75c24df481097464fb1957b76b4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a33368758b4a86a88156168ef4c6ec" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 43741 } } peers { permanent_uuid: "c71ce75c24df481097464fb1957b76b4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 46097 } } peers { permanent_uuid: "6c43506880b44e55b9fdfefbb44f2845" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 34689 } }
I20260619 14:34:45.488317 16442 raft_consensus.cc:729] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P c71ce75c24df481097464fb1957b76b4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c71ce75c24df481097464fb1957b76b4, State: Initialized, Role: FOLLOWER
I20260619 14:34:45.488828 16442 consensus_queue.cc:260] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P c71ce75c24df481097464fb1957b76b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a33368758b4a86a88156168ef4c6ec" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 43741 } } peers { permanent_uuid: "c71ce75c24df481097464fb1957b76b4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 46097 } } peers { permanent_uuid: "6c43506880b44e55b9fdfefbb44f2845" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 34689 } }
I20260619 14:34:45.490044 16442 ts_tablet_manager.cc:1428] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P c71ce75c24df481097464fb1957b76b4: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:34:45.490171 16362 ts_manager.cc:194] Registered new tserver with Master: c71ce75c24df481097464fb1957b76b4 (127.2.2.1:46097)
I20260619 14:34:45.494093 16362 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.2.1@KRBTEST.COM'} at 127.2.2.1:35149
I20260619 14:34:45.496729 16550 heartbeater.cc:503] Master 127.2.2.62:46797 was elected leader, sending a full tablet report...
Jun 19 14:34:45 dist-test-slave-70rm krb5kdc[15780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879685, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.62@KRBTEST.COM for kudu/127.2.2.1@KRBTEST.COM
I20260619 14:34:45.504688 16558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:45.493262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34882 (local address 127.2.2.1:46097)
0619 14:34:45.494339 (+ 1077us) server_negotiation.cc:206] Beginning negotiation
0619 14:34:45.494351 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:34:45.494374 (+ 23us) server_negotiation.cc:407] Connection header received
0619 14:34:45.494445 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:34:45.494449 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:34:45.494644 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:34:45.494785 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:34:45.495562 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:45.496624 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:45.497840 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:45.498482 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:45.498572 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:45.501052 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:34:45.501061 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:34:45.501071 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:34:45.501094 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:34:45.502756 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:45.503294 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:45.503298 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:45.503299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:45.503375 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:45.503670 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:45.503675 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:45.503677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:45.503959 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:34:45.504053 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 14:34:45.504417 (+ 364us) server_negotiation.cc:299] Negotiation successful
0619 14:34:45.504557 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":981,"thread_start_us":95,"threads_started":1}
I20260619 14:34:45.504722 16557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:45.493193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.1:46097 (local address 127.0.0.1:34882)
0619 14:34:45.493673 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:45.493722 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 14:34:45.493881 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:34:45.494815 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:34:45.494820 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:34:45.495256 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:34:45.495414 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:45.495422 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:45.496750 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:45.496754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:45.497708 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:45.497720 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:45.498573 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:45.498590 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:45.498701 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:45.499246 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:34:45.499260 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:34:45.500895 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:34:45.502867 (+ 1972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:34:45.502879 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:34:45.502884 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:34:45.503186 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:34:45.503475 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:34:45.503478 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:34:45.503480 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:34:45.503574 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:34:45.504077 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:34:45.504083 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:34:45.504209 (+ 126us) client_negotiation.cc:769] Sending connection context
0619 14:34:45.504350 (+ 141us) client_negotiation.cc:240] Negotiation successful
0619 14:34:45.504558 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":132,"threads_started":1}
I20260619 14:34:45.505733 16484 tablet_service.cc:1514] Processing DeleteTablet for tablet 388c90dd7e9e4ffc8d551ffc274d0ff7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:34:45 UTC) from {username='kudu', principal='kudu/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:34882
I20260619 14:34:45.506773 16559 tablet_replica.cc:330] stopping tablet replica
I20260619 14:34:45.507062 16559 raft_consensus.cc:2229] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P c71ce75c24df481097464fb1957b76b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:34:45.507308 16559 raft_consensus.cc:2258] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P c71ce75c24df481097464fb1957b76b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:34:45.508250 16559 ts_tablet_manager.cc:1905] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P c71ce75c24df481097464fb1957b76b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:34:45.511718 16559 ts_tablet_manager.cc:1918] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P c71ce75c24df481097464fb1957b76b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:34:45.511832 16559 log.cc:1196] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P c71ce75c24df481097464fb1957b76b4: Deleting WAL directory at /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-0/wal/wals/388c90dd7e9e4ffc8d551ffc274d0ff7
I20260619 14:34:45.512106 16559 ts_tablet_manager.cc:1939] T 388c90dd7e9e4ffc8d551ffc274d0ff7 P c71ce75c24df481097464fb1957b76b4: Deleting consensus metadata
I20260619 14:34:45.513733 16347 catalog_manager.cc:4841] TS c71ce75c24df481097464fb1957b76b4 (127.2.2.1:46097): tablet 388c90dd7e9e4ffc8d551ffc274d0ff7 (table test-table [id=addc849ecca84013828f726f5c546448]) successfully deleted
W20260619 14:34:45.597517 16554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:34:45.597771 16554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:34:45.597851 16554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:34:45.601961 16554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:34:45.602022 16554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:34:45.602133 16554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.2
I20260619 14:34:45.607530 16554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:39547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab
--principal=kudu/127.2.2.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.2:43741
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.2.2
--webserver_port=44979
--tserver_master_addrs=127.2.2.62:46797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.2
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:34:45.608628 16554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:34:45.609683 16554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:34:45.620018 16567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:34:45.620018 16564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:34:45.620136 16554 server_base.cc:1034] running on GCE node
W20260619 14:34:45.620023 16565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:34:45.620927 16554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:34:45.621670 16554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:34:45.622888 16554 hybrid_clock.cc:648] HybridClock initialized: now 1781879685622870 us; error 34 us; skew 500 ppm
Jun 19 14:34:45 dist-test-slave-70rm krb5kdc[15780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879685, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:34:45.625643 16554 init.cc:377] Logged in from keytab as kudu/127.2.2.2@KRBTEST.COM (short username kudu)
I20260619 14:34:45.626677 16554 webserver.cc:460] Webserver started at http://127.2.2.2:44979/ using document root <none> and password file <none>
I20260619 14:34:45.627259 16554 fs_manager.cc:359] Metadata directory not provided
I20260619 14:34:45.627373 16554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:34:45.631305 16554 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260619 14:34:45.633490 16574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:34:45.634835 16554 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 14:34:45.634987 16554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-1/wal
uuid: "55a33368758b4a86a88156168ef4c6ec"
format_stamp: "Formatted at 2026-06-19 14:34:44 on dist-test-slave-70rm"
server_key: "809f69815f8eca585e8ea3aca958ee8b"
server_key_iv: "732a1081bc3a6ee3233fb5a1cdb2f226"
server_key_version: "encryptionkey@0"
I20260619 14:34:45.635565 16554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:34:45.654232 16554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:34:45.657588 16554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:34:45.657828 16554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:34:45.658571 16554 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:34:45.659478 16554 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:34:45.659552 16554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:34:45.659626 16554 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:34:45.659668 16554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:34:45.669520 16554 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.2:43741
I20260619 14:34:45.669543 16687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.2:43741 every 8 connection(s)
I20260619 14:34:45.670653 16554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
I20260619 14:34:45.678746 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 16554
I20260619 14:34:45.679199 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.3:34689
--local_ip_for_outbound_sockets=127.2.2.3
--tserver_master_addrs=127.2.2.62:46797
--webserver_port=45009
--webserver_interface=127.2.2.3
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab
--principal=kudu/127.2.2.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:39547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:34:45 dist-test-slave-70rm krb5kdc[15780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879685, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.2@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:34:45.686033 16552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:45.672700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:51551 (local address 127.2.2.62:46797)
0619 14:34:45.672864 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 14:34:45.672868 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:34:45.673919 (+ 1051us) server_negotiation.cc:407] Connection header received
0619 14:34:45.674959 (+ 1040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:34:45.674962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:34:45.675035 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:34:45.675140 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:34:45.676499 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:45.677085 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:45.678358 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:45.678604 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:45.678721 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:45.681415 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:34:45.681419 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:34:45.681424 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:34:45.681460 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:34:45.683172 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:45.683879 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:45.683882 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:45.683884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:45.683966 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:45.684523 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:45.684526 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:45.684527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:45.684819 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:34:45.684922 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 14:34:45.685702 (+ 780us) server_negotiation.cc:299] Negotiation successful
0619 14:34:45.685878 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 14:34:45.687461 16690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:45.673023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46797 (local address 127.2.2.2:51551)
0619 14:34:45.673741 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:45.673785 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 14:34:45.674687 (+ 902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:34:45.675331 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:34:45.675338 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:34:45.675819 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:34:45.676337 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:45.676350 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:45.677227 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:45.677231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:45.678230 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:45.678241 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:45.678705 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:45.678774 (+ 69us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:45.679005 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:45.679676 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:34:45.679700 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:34:45.681198 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:34:45.683306 (+ 2108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:34:45.683315 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:34:45.683335 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:34:45.683754 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:34:45.684065 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:34:45.684070 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:34:45.684073 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:34:45.684292 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:34:45.684937 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:34:45.684947 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:34:45.685366 (+ 419us) client_negotiation.cc:769] Sending connection context
0619 14:34:45.685660 (+ 294us) client_negotiation.cc:240] Negotiation successful
0619 14:34:45.686109 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":128,"threads_started":1}
I20260619 14:34:45.688879 16688 heartbeater.cc:347] Connected to a master server at 127.2.2.62:46797
I20260619 14:34:45.689208 16688 heartbeater.cc:464] Registering TS with master...
I20260619 14:34:45.689918 16688 heartbeater.cc:511] Master 127.2.2.62:46797 requested a full tablet report, sending...
I20260619 14:34:45.691272 16362 ts_manager.cc:194] Registered new tserver with Master: 55a33368758b4a86a88156168ef4c6ec (127.2.2.2:43741)
I20260619 14:34:45.691890 16362 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.2.2@KRBTEST.COM'} at 127.2.2.2:51551
W20260619 14:34:45.794797 16691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:34:45.795089 16691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:34:45.795162 16691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:34:45.799396 16691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:34:45.799465 16691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:34:45.799566 16691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.3
I20260619 14:34:45.805091 16691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:39547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab
--principal=kudu/127.2.2.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.3:34689
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.2.3
--webserver_port=45009
--tserver_master_addrs=127.2.2.62:46797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.3
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:34:45.806280 16691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:34:45.807319 16691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:34:45.817494 16697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:34:45.817523 16696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:34:45.817544 16699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:34:45.817644 16691 server_base.cc:1034] running on GCE node
I20260619 14:34:45.818281 16691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:34:45.818925 16691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:34:45.820125 16691 hybrid_clock.cc:648] HybridClock initialized: now 1781879685820094 us; error 34 us; skew 500 ppm
Jun 19 14:34:45 dist-test-slave-70rm krb5kdc[15780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879685, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:34:45.823026 16691 init.cc:377] Logged in from keytab as kudu/127.2.2.3@KRBTEST.COM (short username kudu)
I20260619 14:34:45.823988 16691 webserver.cc:460] Webserver started at http://127.2.2.3:45009/ using document root <none> and password file <none>
I20260619 14:34:45.824559 16691 fs_manager.cc:359] Metadata directory not provided
I20260619 14:34:45.824661 16691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:34:45.828712 16691 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:34:45.830986 16706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:34:45.832257 16691 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:34:45.832381 16691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-2/wal
uuid: "6c43506880b44e55b9fdfefbb44f2845"
format_stamp: "Formatted at 2026-06-19 14:34:44 on dist-test-slave-70rm"
server_key: "82a6f7b11a093d9d44720b057201e669"
server_key_iv: "fb581898a9f147bf76482f2eb9f83942"
server_key_version: "encryptionkey@0"
I20260619 14:34:45.832928 16691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:34:45.854576 16691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:34:45.857836 16691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:34:45.858057 16691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:34:45.858704 16691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:34:45.859571 16691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:34:45.859640 16691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:34:45.859714 16691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:34:45.859754 16691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:34:45.868803 16691 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.3:34689
I20260619 14:34:45.868826 16819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.3:34689 every 8 connection(s)
I20260619 14:34:45.869853 16691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
I20260619 14:34:45.876564 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 16691
Jun 19 14:34:45 dist-test-slave-70rm krb5kdc[15780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879685, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.3@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:34:45.885031 16552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:45.871777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.3:57719 (local address 127.2.2.62:46797)
0619 14:34:45.871926 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 14:34:45.871930 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:34:45.873043 (+ 1113us) server_negotiation.cc:407] Connection header received
0619 14:34:45.873914 (+ 871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:34:45.873916 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:34:45.873964 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:34:45.874042 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:34:45.875405 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:45.875969 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:45.877831 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:45.878189 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:45.878305 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:45.880858 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:34:45.880863 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:34:45.880866 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:34:45.880894 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:34:45.882446 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:45.883143 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:45.883147 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:45.883150 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:45.883247 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:45.883690 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:45.883693 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:45.883695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:45.883910 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:34:45.884038 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 14:34:45.884689 (+ 651us) server_negotiation.cc:299] Negotiation successful
0619 14:34:45.884834 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 14:34:45.885823 16824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:45.877378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38880 (local address 127.2.2.62:46797)
0619 14:34:45.877650 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0619 14:34:45.877655 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:34:45.877766 (+ 111us) server_negotiation.cc:407] Connection header received
0619 14:34:45.877922 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:34:45.877924 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:34:45.877967 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:34:45.878044 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:34:45.878891 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:45.879407 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:45.880253 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:45.880541 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:45.880651 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:45.882294 (+ 1643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:34:45.882297 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:34:45.882299 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:34:45.882332 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:34:45.883937 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:45.884427 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:45.884430 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:45.884432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:45.884523 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:45.884843 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:45.884846 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:45.884847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:45.885056 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:34:45.885242 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0619 14:34:45.885586 (+ 344us) server_negotiation.cc:299] Negotiation successful
0619 14:34:45.885680 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":86,"threads_started":1}
I20260619 14:34:45.886107 16822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:45.872131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46797 (local address 127.2.2.3:57719)
0619 14:34:45.872876 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:45.872921 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 14:34:45.873721 (+ 800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:34:45.874208 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:34:45.874215 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:34:45.874614 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:34:45.875196 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:45.875213 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:45.876214 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:45.876220 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:45.877655 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:45.877669 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:45.878292 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:45.878344 (+ 52us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:45.878499 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:45.879152 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:34:45.879170 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:34:45.880696 (+ 1526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:34:45.882558 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:34:45.882567 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:34:45.882588 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:34:45.883031 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:34:45.883349 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:34:45.883352 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:34:45.883354 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:34:45.883559 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:34:45.884046 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:34:45.884055 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:34:45.884364 (+ 309us) client_negotiation.cc:769] Sending connection context
0619 14:34:45.884598 (+ 234us) client_negotiation.cc:240] Negotiation successful
0619 14:34:45.884961 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":112,"threads_started":1}
I20260619 14:34:45.887347 16820 heartbeater.cc:347] Connected to a master server at 127.2.2.62:46797
I20260619 14:34:45.887602 16820 heartbeater.cc:464] Registering TS with master...
I20260619 14:34:45.888139 16820 heartbeater.cc:511] Master 127.2.2.62:46797 requested a full tablet report, sending...
I20260619 14:34:45.889151 16362 ts_manager.cc:194] Registered new tserver with Master: 6c43506880b44e55b9fdfefbb44f2845 (127.2.2.3:34689)
I20260619 14:34:45.889700 16362 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.2.3@KRBTEST.COM'} at 127.2.2.3:57719
I20260619 14:34:45.900051 2056 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:34:45.904798 16824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:45.900753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38896 (local address 127.2.2.62:46797)
0619 14:34:45.900903 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 14:34:45.900907 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:34:45.901104 (+ 197us) server_negotiation.cc:407] Connection header received
0619 14:34:45.901215 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:34:45.901217 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:34:45.901327 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:34:45.901429 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:34:45.901681 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:45.902299 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:45.903209 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:45.903448 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:45.903546 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:45.903987 (+ 441us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:34:45.904411 (+ 424us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:34:45.904518 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 14:34:45.904618 (+ 100us) server_negotiation.cc:299] Negotiation successful
0619 14:34:45.904681 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 14:34:45.905534 16362 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:38896:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:34:45.908885 16362 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:34:45.916483 16484 tablet_service.cc:1467] Processing CreateTablet for tablet b205a6a6b973472097481a8ee9b9e588 (DEFAULT_TABLE table=test-table [id=69de8017f2044128947d1b98edf1700a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:34:45.917201 16484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b205a6a6b973472097481a8ee9b9e588. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:34:45.920147 16442 tablet_bootstrap.cc:492] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4: Bootstrap starting.
I20260619 14:34:45.921008 16557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:45.915452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.2:43741 (local address 127.0.0.1:55266)
0619 14:34:45.915621 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:45.915637 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:34:45.915720 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:34:45.916591 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:34:45.916594 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:34:45.916612 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:34:45.916722 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:45.916729 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:45.918222 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:45.918227 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:45.919270 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:45.919280 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:45.920518 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:45.920521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:45.920631 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:45.920691 (+ 60us) client_negotiation.cc:769] Sending connection context
0619 14:34:45.920750 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 14:34:45.920842 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"mutex_wait_us":1}
I20260619 14:34:45.921171 16442 tablet_bootstrap.cc:654] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4: Neither blocks nor log segments found. Creating new log.
I20260619 14:34:45.921267 16831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:45.916194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53518 (local address 127.2.2.3:34689)
0619 14:34:45.916528 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0619 14:34:45.916541 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:34:45.916562 (+ 21us) server_negotiation.cc:407] Connection header received
0619 14:34:45.916630 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:34:45.916635 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:34:45.916868 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:34:45.917033 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:34:45.917270 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:45.918640 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:45.919860 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:45.920669 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:45.920776 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:45.920924 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 14:34:45.920981 (+ 57us) server_negotiation.cc:299] Negotiation successful
0619 14:34:45.921101 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":121,"threads_started":1}
I20260619 14:34:45.921743 16829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:45.915679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55266 (local address 127.2.2.2:43741)
0619 14:34:45.916038 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0619 14:34:45.916051 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:34:45.916076 (+ 25us) server_negotiation.cc:407] Connection header received
0619 14:34:45.916161 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:34:45.916169 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:34:45.916424 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:34:45.916600 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:34:45.916849 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:45.918044 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:45.919407 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:45.920371 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:45.921097 (+ 726us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:45.921263 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 14:34:45.921385 (+ 122us) server_negotiation.cc:299] Negotiation successful
0619 14:34:45.921495 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":170,"threads_started":1}
I20260619 14:34:45.921763 16830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:45.915768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.3:34689 (local address 127.0.0.1:53518)
0619 14:34:45.916143 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:45.916156 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:34:45.916236 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:34:45.917011 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:34:45.917013 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:34:45.917031 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:34:45.917123 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:45.917129 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:45.918784 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:45.918787 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:45.919740 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:45.919748 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:45.920775 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:45.920778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:45.920857 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:45.920877 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 14:34:45.920935 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 14:34:45.920991 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":123,"threads_started":1}
I20260619 14:34:45.922853 16442 tablet_bootstrap.cc:492] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4: No bootstrap required, opened a new log
I20260619 14:34:45.922957 16442 ts_tablet_manager.cc:1397] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 14:34:45.923368 16442 raft_consensus.cc:348] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a33368758b4a86a88156168ef4c6ec" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 43741 } } peers { permanent_uuid: "6c43506880b44e55b9fdfefbb44f2845" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 34689 } } peers { permanent_uuid: "c71ce75c24df481097464fb1957b76b4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 46097 } }
I20260619 14:34:45.923501 16442 raft_consensus.cc:374] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:34:45.923553 16442 raft_consensus.cc:729] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c71ce75c24df481097464fb1957b76b4, State: Initialized, Role: FOLLOWER
I20260619 14:34:45.923681 16442 consensus_queue.cc:260] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a33368758b4a86a88156168ef4c6ec" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 43741 } } peers { permanent_uuid: "6c43506880b44e55b9fdfefbb44f2845" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 34689 } } peers { permanent_uuid: "c71ce75c24df481097464fb1957b76b4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 46097 } }
I20260619 14:34:45.924517 16442 ts_tablet_manager.cc:1428] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260619 14:34:45.924964 16622 tablet_service.cc:1467] Processing CreateTablet for tablet b205a6a6b973472097481a8ee9b9e588 (DEFAULT_TABLE table=test-table [id=69de8017f2044128947d1b98edf1700a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:34:45.924964 16754 tablet_service.cc:1467] Processing CreateTablet for tablet b205a6a6b973472097481a8ee9b9e588 (DEFAULT_TABLE table=test-table [id=69de8017f2044128947d1b98edf1700a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:34:45.926102 16622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b205a6a6b973472097481a8ee9b9e588. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:34:45.926105 16754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b205a6a6b973472097481a8ee9b9e588. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:34:45.934782 16834 tablet_bootstrap.cc:492] T b205a6a6b973472097481a8ee9b9e588 P 55a33368758b4a86a88156168ef4c6ec: Bootstrap starting.
I20260619 14:34:45.934782 16833 tablet_bootstrap.cc:492] T b205a6a6b973472097481a8ee9b9e588 P 6c43506880b44e55b9fdfefbb44f2845: Bootstrap starting.
I20260619 14:34:45.937929 16834 tablet_bootstrap.cc:654] T b205a6a6b973472097481a8ee9b9e588 P 55a33368758b4a86a88156168ef4c6ec: Neither blocks nor log segments found. Creating new log.
I20260619 14:34:45.938151 16833 tablet_bootstrap.cc:654] T b205a6a6b973472097481a8ee9b9e588 P 6c43506880b44e55b9fdfefbb44f2845: Neither blocks nor log segments found. Creating new log.
I20260619 14:34:45.939033 16833 log.cc:824] T b205a6a6b973472097481a8ee9b9e588 P 6c43506880b44e55b9fdfefbb44f2845: Log is configured to *not* fsync() on all Append() calls
I20260619 14:34:45.939033 16834 log.cc:824] T b205a6a6b973472097481a8ee9b9e588 P 55a33368758b4a86a88156168ef4c6ec: Log is configured to *not* fsync() on all Append() calls
I20260619 14:34:45.941396 16833 tablet_bootstrap.cc:492] T b205a6a6b973472097481a8ee9b9e588 P 6c43506880b44e55b9fdfefbb44f2845: No bootstrap required, opened a new log
I20260619 14:34:45.941396 16834 tablet_bootstrap.cc:492] T b205a6a6b973472097481a8ee9b9e588 P 55a33368758b4a86a88156168ef4c6ec: No bootstrap required, opened a new log
I20260619 14:34:45.941612 16833 ts_tablet_manager.cc:1397] T b205a6a6b973472097481a8ee9b9e588 P 6c43506880b44e55b9fdfefbb44f2845: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 14:34:45.941612 16834 ts_tablet_manager.cc:1397] T b205a6a6b973472097481a8ee9b9e588 P 55a33368758b4a86a88156168ef4c6ec: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260619 14:34:45.945758 16833 raft_consensus.cc:348] T b205a6a6b973472097481a8ee9b9e588 P 6c43506880b44e55b9fdfefbb44f2845 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a33368758b4a86a88156168ef4c6ec" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 43741 } } peers { permanent_uuid: "6c43506880b44e55b9fdfefbb44f2845" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 34689 } } peers { permanent_uuid: "c71ce75c24df481097464fb1957b76b4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 46097 } }
I20260619 14:34:45.945758 16834 raft_consensus.cc:348] T b205a6a6b973472097481a8ee9b9e588 P 55a33368758b4a86a88156168ef4c6ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a33368758b4a86a88156168ef4c6ec" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 43741 } } peers { permanent_uuid: "6c43506880b44e55b9fdfefbb44f2845" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 34689 } } peers { permanent_uuid: "c71ce75c24df481097464fb1957b76b4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 46097 } }
I20260619 14:34:45.946096 16834 raft_consensus.cc:374] T b205a6a6b973472097481a8ee9b9e588 P 55a33368758b4a86a88156168ef4c6ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:34:45.946096 16833 raft_consensus.cc:374] T b205a6a6b973472097481a8ee9b9e588 P 6c43506880b44e55b9fdfefbb44f2845 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:34:45.946177 16834 raft_consensus.cc:729] T b205a6a6b973472097481a8ee9b9e588 P 55a33368758b4a86a88156168ef4c6ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55a33368758b4a86a88156168ef4c6ec, State: Initialized, Role: FOLLOWER
I20260619 14:34:45.946198 16833 raft_consensus.cc:729] T b205a6a6b973472097481a8ee9b9e588 P 6c43506880b44e55b9fdfefbb44f2845 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c43506880b44e55b9fdfefbb44f2845, State: Initialized, Role: FOLLOWER
I20260619 14:34:45.946713 16834 consensus_queue.cc:260] T b205a6a6b973472097481a8ee9b9e588 P 55a33368758b4a86a88156168ef4c6ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a33368758b4a86a88156168ef4c6ec" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 43741 } } peers { permanent_uuid: "6c43506880b44e55b9fdfefbb44f2845" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 34689 } } peers { permanent_uuid: "c71ce75c24df481097464fb1957b76b4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 46097 } }
I20260619 14:34:45.946713 16833 consensus_queue.cc:260] T b205a6a6b973472097481a8ee9b9e588 P 6c43506880b44e55b9fdfefbb44f2845 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a33368758b4a86a88156168ef4c6ec" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 43741 } } peers { permanent_uuid: "6c43506880b44e55b9fdfefbb44f2845" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 34689 } } peers { permanent_uuid: "c71ce75c24df481097464fb1957b76b4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 46097 } }
I20260619 14:34:45.947448 16688 heartbeater.cc:503] Master 127.2.2.62:46797 was elected leader, sending a full tablet report...
I20260619 14:34:45.947674 16820 heartbeater.cc:503] Master 127.2.2.62:46797 was elected leader, sending a full tablet report...
I20260619 14:34:45.947837 16833 ts_tablet_manager.cc:1428] T b205a6a6b973472097481a8ee9b9e588 P 6c43506880b44e55b9fdfefbb44f2845: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 14:34:45.948222 16834 ts_tablet_manager.cc:1428] T b205a6a6b973472097481a8ee9b9e588 P 55a33368758b4a86a88156168ef4c6ec: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
W20260619 14:34:45.966390 16551 tablet.cc:2369] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:34:46.108492 16556 raft_consensus.cc:482] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:34:46.108721 16556 raft_consensus.cc:504] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a33368758b4a86a88156168ef4c6ec" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 43741 } } peers { permanent_uuid: "6c43506880b44e55b9fdfefbb44f2845" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 34689 } } peers { permanent_uuid: "c71ce75c24df481097464fb1957b76b4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 46097 } }
I20260619 14:34:46.109820 16556 leader_election.cc:288] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 55a33368758b4a86a88156168ef4c6ec (127.2.2.2:43741), 6c43506880b44e55b9fdfefbb44f2845 (127.2.2.3:34689)
I20260619 14:34:46.114377 16831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:46.110264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:44169 (local address 127.2.2.3:34689)
0619 14:34:46.110399 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 14:34:46.110403 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:34:46.110591 (+ 188us) server_negotiation.cc:407] Connection header received
0619 14:34:46.110761 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:34:46.110765 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:34:46.110841 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:34:46.110922 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:34:46.111359 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:46.111900 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:46.113087 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:46.113656 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:46.113783 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:46.113850 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 14:34:46.114158 (+ 308us) server_negotiation.cc:299] Negotiation successful
0619 14:34:46.114254 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 14:34:46.114416 16829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:46.110223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:39553 (local address 127.2.2.2:43741)
0619 14:34:46.110387 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 14:34:46.110394 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:34:46.110872 (+ 478us) server_negotiation.cc:407] Connection header received
0619 14:34:46.111053 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:34:46.111056 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:34:46.111103 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:34:46.111185 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:34:46.111475 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:46.112021 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:46.113191 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:46.113799 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:46.113905 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:46.113968 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 14:34:46.114236 (+ 268us) server_negotiation.cc:299] Negotiation successful
0619 14:34:46.114311 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 14:34:46.114398 16839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:46.110096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.3:34689 (local address 127.2.2.1:44169)
0619 14:34:46.110438 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:46.110454 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:34:46.110596 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:34:46.111052 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:34:46.111055 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:34:46.111076 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:34:46.111229 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:46.111238 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:46.112026 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:46.112030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:46.112956 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:46.112965 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:46.113790 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:46.113794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:46.113921 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:46.113983 (+ 62us) client_negotiation.cc:769] Sending connection context
0619 14:34:46.114110 (+ 127us) client_negotiation.cc:240] Negotiation successful
0619 14:34:46.114223 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":22,"thread_start_us":111,"threads_started":1}
I20260619 14:34:46.114851 16840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:46.110098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.2:43741 (local address 127.2.2.1:39553)
0619 14:34:46.110747 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:46.110760 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:34:46.110863 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:34:46.111211 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:34:46.111213 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:34:46.111230 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:34:46.111334 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:46.111342 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:46.112137 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:46.112139 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:46.112969 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:46.112984 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:46.113927 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:46.113930 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:46.114030 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:46.114058 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 14:34:46.114597 (+ 539us) client_negotiation.cc:240] Negotiation successful
0619 14:34:46.114682 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"mutex_wait_us":338,"thread_start_us":98,"threads_started":1}
I20260619 14:34:46.115097 16774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b205a6a6b973472097481a8ee9b9e588" candidate_uuid: "c71ce75c24df481097464fb1957b76b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c43506880b44e55b9fdfefbb44f2845" is_pre_election: true
I20260619 14:34:46.115461 16774 raft_consensus.cc:2454] T b205a6a6b973472097481a8ee9b9e588 P 6c43506880b44e55b9fdfefbb44f2845 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c71ce75c24df481097464fb1957b76b4 in term 0.
I20260619 14:34:46.115520 16642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b205a6a6b973472097481a8ee9b9e588" candidate_uuid: "c71ce75c24df481097464fb1957b76b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55a33368758b4a86a88156168ef4c6ec" is_pre_election: true
I20260619 14:34:46.115772 16642 raft_consensus.cc:2454] T b205a6a6b973472097481a8ee9b9e588 P 55a33368758b4a86a88156168ef4c6ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c71ce75c24df481097464fb1957b76b4 in term 0.
I20260619 14:34:46.116003 16439 leader_election.cc:302] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c43506880b44e55b9fdfefbb44f2845, c71ce75c24df481097464fb1957b76b4; no voters:
I20260619 14:34:46.116317 16556 raft_consensus.cc:2781] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:34:46.116418 16556 raft_consensus.cc:482] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:34:46.116463 16556 raft_consensus.cc:3037] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:34:46.117507 16556 raft_consensus.cc:504] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a33368758b4a86a88156168ef4c6ec" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 43741 } } peers { permanent_uuid: "6c43506880b44e55b9fdfefbb44f2845" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 34689 } } peers { permanent_uuid: "c71ce75c24df481097464fb1957b76b4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 46097 } }
I20260619 14:34:46.117869 16556 leader_election.cc:288] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4 [CANDIDATE]: Term 1 election: Requested vote from peers 55a33368758b4a86a88156168ef4c6ec (127.2.2.2:43741), 6c43506880b44e55b9fdfefbb44f2845 (127.2.2.3:34689)
I20260619 14:34:46.118325 16642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b205a6a6b973472097481a8ee9b9e588" candidate_uuid: "c71ce75c24df481097464fb1957b76b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55a33368758b4a86a88156168ef4c6ec"
I20260619 14:34:46.118325 16774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b205a6a6b973472097481a8ee9b9e588" candidate_uuid: "c71ce75c24df481097464fb1957b76b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c43506880b44e55b9fdfefbb44f2845"
I20260619 14:34:46.118466 16774 raft_consensus.cc:3037] T b205a6a6b973472097481a8ee9b9e588 P 6c43506880b44e55b9fdfefbb44f2845 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:34:46.118466 16642 raft_consensus.cc:3037] T b205a6a6b973472097481a8ee9b9e588 P 55a33368758b4a86a88156168ef4c6ec [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:34:46.119867 16774 raft_consensus.cc:2454] T b205a6a6b973472097481a8ee9b9e588 P 6c43506880b44e55b9fdfefbb44f2845 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c71ce75c24df481097464fb1957b76b4 in term 1.
I20260619 14:34:46.119867 16642 raft_consensus.cc:2454] T b205a6a6b973472097481a8ee9b9e588 P 55a33368758b4a86a88156168ef4c6ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c71ce75c24df481097464fb1957b76b4 in term 1.
I20260619 14:34:46.120186 16438 leader_election.cc:302] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55a33368758b4a86a88156168ef4c6ec, c71ce75c24df481097464fb1957b76b4; no voters:
I20260619 14:34:46.120395 16556 raft_consensus.cc:2781] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:34:46.120604 16556 raft_consensus.cc:686] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4 [term 1 LEADER]: Becoming Leader. State: Replica: c71ce75c24df481097464fb1957b76b4, State: Running, Role: LEADER
W20260619 14:34:46.120850 16821 tablet.cc:2369] T b205a6a6b973472097481a8ee9b9e588 P 6c43506880b44e55b9fdfefbb44f2845: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:34:46.120904 16556 consensus_queue.cc:237] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a33368758b4a86a88156168ef4c6ec" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 43741 } } peers { permanent_uuid: "6c43506880b44e55b9fdfefbb44f2845" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 34689 } } peers { permanent_uuid: "c71ce75c24df481097464fb1957b76b4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 46097 } }
I20260619 14:34:46.124100 16360 catalog_manager.cc:5495] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to c71ce75c24df481097464fb1957b76b4 (127.2.2.1). New cstate: current_term: 1 leader_uuid: "c71ce75c24df481097464fb1957b76b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a33368758b4a86a88156168ef4c6ec" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 43741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c43506880b44e55b9fdfefbb44f2845" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 34689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c71ce75c24df481097464fb1957b76b4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 46097 } health_report { overall_health: HEALTHY } } }
I20260619 14:34:46.168788 16844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:46.164946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34898 (local address 127.2.2.1:46097)
0619 14:34:46.165247 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0619 14:34:46.165254 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:34:46.165266 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:34:46.165338 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:34:46.165340 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:34:46.165392 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:34:46.165529 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:34:46.165784 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:46.166387 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:46.167289 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:46.167529 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:46.167642 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:46.167923 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:34:46.168376 (+ 453us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:34:46.168517 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 14:34:46.168554 (+ 37us) server_negotiation.cc:299] Negotiation successful
0619 14:34:46.168614 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":109,"threads_started":1}
W20260619 14:34:46.171872 16689 tablet.cc:2369] T b205a6a6b973472097481a8ee9b9e588 P 55a33368758b4a86a88156168ef4c6ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:34:46.172998 16642 raft_consensus.cc:1261] T b205a6a6b973472097481a8ee9b9e588 P 55a33368758b4a86a88156168ef4c6ec [term 1 FOLLOWER]: Refusing update from remote peer c71ce75c24df481097464fb1957b76b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:34:46.172999 16774 raft_consensus.cc:1261] T b205a6a6b973472097481a8ee9b9e588 P 6c43506880b44e55b9fdfefbb44f2845 [term 1 FOLLOWER]: Refusing update from remote peer c71ce75c24df481097464fb1957b76b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:34:46.173736 16556 consensus_queue.cc:1035] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c43506880b44e55b9fdfefbb44f2845" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 34689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:34:46.173943 16841 consensus_queue.cc:1035] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55a33368758b4a86a88156168ef4c6ec" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 43741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:34:46.181376 16845 mvcc.cc:205] Tried to move back new op lower bound from 7298579194557952000 to 7298579194354335744. Current Snapshot: MvccSnapshot[applied={T|T < 7298579194557952000}]
I20260619 14:34:46.184396 16847 mvcc.cc:205] Tried to move back new op lower bound from 7298579194557952000 to 7298579194354335744. Current Snapshot: MvccSnapshot[applied={T|T < 7298579194557952000}]
I20260619 14:34:46.190017 16360 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:38896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:34:46.190248 16360 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:38896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:34:46.192837 16360 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986: Sending DeleteTablet for 3 replicas of tablet b205a6a6b973472097481a8ee9b9e588
I20260619 14:34:46.193586 16484 tablet_service.cc:1514] Processing DeleteTablet for tablet b205a6a6b973472097481a8ee9b9e588 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:34:46 UTC) from {username='kudu', principal='kudu/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:34882
I20260619 14:34:46.193647 16622 tablet_service.cc:1514] Processing DeleteTablet for tablet b205a6a6b973472097481a8ee9b9e588 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:34:46 UTC) from {username='kudu', principal='kudu/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:55266
I20260619 14:34:46.194159 16854 tablet_replica.cc:330] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4: stopping tablet replica
I20260619 14:34:46.194249 16855 tablet_replica.cc:330] T b205a6a6b973472097481a8ee9b9e588 P 55a33368758b4a86a88156168ef4c6ec: stopping tablet replica
I20260619 14:34:46.194260 16754 tablet_service.cc:1514] Processing DeleteTablet for tablet b205a6a6b973472097481a8ee9b9e588 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:34:46 UTC) from {username='kudu', principal='kudu/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:53518
I20260619 14:34:46.194342 16854 raft_consensus.cc:2229] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:34:46.194685 16854 raft_consensus.cc:2258] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:34:46.194705 16855 raft_consensus.cc:2229] T b205a6a6b973472097481a8ee9b9e588 P 55a33368758b4a86a88156168ef4c6ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:34:46.195012 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 16400
I20260619 14:34:46.195075 16855 raft_consensus.cc:2258] T b205a6a6b973472097481a8ee9b9e588 P 55a33368758b4a86a88156168ef4c6ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:34:46.196702 16856 tablet_replica.cc:330] T b205a6a6b973472097481a8ee9b9e588 P 6c43506880b44e55b9fdfefbb44f2845: stopping tablet replica
I20260619 14:34:46.196988 16856 raft_consensus.cc:2229] T b205a6a6b973472097481a8ee9b9e588 P 6c43506880b44e55b9fdfefbb44f2845 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:34:46.197266 16856 raft_consensus.cc:2258] T b205a6a6b973472097481a8ee9b9e588 P 6c43506880b44e55b9fdfefbb44f2845 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:34:46.198714 16856 ts_tablet_manager.cc:1905] T b205a6a6b973472097481a8ee9b9e588 P 6c43506880b44e55b9fdfefbb44f2845: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:34:46.198741 16855 ts_tablet_manager.cc:1905] T b205a6a6b973472097481a8ee9b9e588 P 55a33368758b4a86a88156168ef4c6ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:34:46.200922 16856 ts_tablet_manager.cc:1918] T b205a6a6b973472097481a8ee9b9e588 P 6c43506880b44e55b9fdfefbb44f2845: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:34:46.201053 16856 log.cc:1196] T b205a6a6b973472097481a8ee9b9e588 P 6c43506880b44e55b9fdfefbb44f2845: Deleting WAL directory at /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-2/wal/wals/b205a6a6b973472097481a8ee9b9e588
I20260619 14:34:46.201414 16856 ts_tablet_manager.cc:1939] T b205a6a6b973472097481a8ee9b9e588 P 6c43506880b44e55b9fdfefbb44f2845: Deleting consensus metadata
I20260619 14:34:46.202024 16855 ts_tablet_manager.cc:1918] T b205a6a6b973472097481a8ee9b9e588 P 55a33368758b4a86a88156168ef4c6ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:34:46.202168 16855 log.cc:1196] T b205a6a6b973472097481a8ee9b9e588 P 55a33368758b4a86a88156168ef4c6ec: Deleting WAL directory at /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-1/wal/wals/b205a6a6b973472097481a8ee9b9e588
W20260619 14:34:46.202278 16347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.2.1:46097 (error 108)
I20260619 14:34:46.202513 16855 ts_tablet_manager.cc:1939] T b205a6a6b973472097481a8ee9b9e588 P 55a33368758b4a86a88156168ef4c6ec: Deleting consensus metadata
I20260619 14:34:46.202765 16349 catalog_manager.cc:4841] TS 6c43506880b44e55b9fdfefbb44f2845 (127.2.2.3:34689): tablet b205a6a6b973472097481a8ee9b9e588 (table test-table [id=69de8017f2044128947d1b98edf1700a]) successfully deleted
W20260619 14:34:46.203373 16830 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:34:46.203105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.1:46097 (local address 127.0.0.1:34908)
0619 14:34:46.203230 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:46.203294 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.1:46097: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 14:34:46.203603 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 16554
W20260619 14:34:46.203560 16347 catalog_manager.cc:4584] TS c71ce75c24df481097464fb1957b76b4 (127.2.2.1:46097): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b205a6a6b973472097481a8ee9b9e588: Network error: Client connection negotiation failed: client connection to 127.2.2.1:46097: connect: Connection reset by peer (error 104)
I20260619 14:34:46.209395 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 16691
I20260619 14:34:46.211436 16830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:46.209596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.2:43741 (local address 127.0.0.1:55268)
0619 14:34:46.211248 (+ 1652us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:46.211323 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.2:43741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1538}
I20260619 14:34:46.215127 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 16329
I20260619 14:34:46.221987 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:46797
--webserver_interface=127.2.2.62
--webserver_port=45891
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:39547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.2.62:46797
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:34:46.334631 16858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:34:46.334976 16858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:34:46.335057 16858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:34:46.339207 16858 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:34:46.339294 16858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:34:46.339329 16858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:34:46.339370 16858 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:34:46.339409 16858 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:34:46.344775 16858 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:39547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.2.62:46797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:46797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.2.62
--webserver_port=45891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:34:46.345917 16858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:34:46.347036 16858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:34:46.356371 16864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:34:46.356389 16863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:34:46.356668 16858 server_base.cc:1034] running on GCE node
W20260619 14:34:46.356392 16866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:34:46.357347 16858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:34:46.358599 16858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:34:46.359783 16858 hybrid_clock.cc:648] HybridClock initialized: now 1781879686359769 us; error 29 us; skew 500 ppm
Jun 19 14:34:46 dist-test-slave-70rm krb5kdc[15780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879686, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:34:46.362596 16858 init.cc:377] Logged in from keytab as kudu/127.2.2.62@KRBTEST.COM (short username kudu)
I20260619 14:34:46.363531 16858 webserver.cc:460] Webserver started at http://127.2.2.62:45891/ using document root <none> and password file <none>
I20260619 14:34:46.364107 16858 fs_manager.cc:359] Metadata directory not provided
I20260619 14:34:46.364200 16858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:34:46.368052 16858 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:34:46.370241 16873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:34:46.371527 16858 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:34:46.371667 16858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/master-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/master-0/wal
uuid: "f971ee13217e41268e22d92d2a111986"
format_stamp: "Formatted at 2026-06-19 14:34:43 on dist-test-slave-70rm"
server_key: "4e56bd85d38ea569d4be8f50d6aaa9ad"
server_key_iv: "d022057fa1f55ec60b11a6bb14e89137"
server_key_version: "encryptionkey@0"
I20260619 14:34:46.372221 16858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:34:46.398077 16858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:34:46.401651 16858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:34:46.401867 16858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:34:46.409152 16858 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.62:46797
I20260619 14:34:46.409471 16925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.62:46797 every 8 connection(s)
I20260619 14:34:46.410357 16858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
I20260619 14:34:46.414561 16926 sys_catalog.cc:263] Verifying existing consensus state
I20260619 14:34:46.416128 16926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986: Bootstrap starting.
I20260619 14:34:46.418732 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 16858
I20260619 14:34:46.419217 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.1:46097
--local_ip_for_outbound_sockets=127.2.2.1
--tserver_master_addrs=127.2.2.62:46797
--webserver_port=38943
--webserver_interface=127.2.2.1
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab
--principal=kudu/127.2.2.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:39547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:34:46.434831 16926 log.cc:824] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986: Log is configured to *not* fsync() on all Append() calls
I20260619 14:34:46.446086 16926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:34:46.446561 16926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986: Bootstrap complete.
I20260619 14:34:46.450765 16926 raft_consensus.cc:348] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f971ee13217e41268e22d92d2a111986" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 46797 } }
I20260619 14:34:46.451191 16926 raft_consensus.cc:729] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f971ee13217e41268e22d92d2a111986, State: Initialized, Role: FOLLOWER
I20260619 14:34:46.451689 16926 consensus_queue.cc:260] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f971ee13217e41268e22d92d2a111986" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 46797 } }
I20260619 14:34:46.451804 16926 raft_consensus.cc:388] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:34:46.451906 16926 raft_consensus.cc:482] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:34:46.452025 16926 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986 [term 2 FOLLOWER]: Advancing to term 3
I20260619 14:34:46.454039 16926 raft_consensus.cc:504] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f971ee13217e41268e22d92d2a111986" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 46797 } }
I20260619 14:34:46.454434 16926 leader_election.cc:302] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f971ee13217e41268e22d92d2a111986; no voters:
I20260619 14:34:46.454797 16926 leader_election.cc:288] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 14:34:46.454886 16930 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 14:34:46.455077 16930 raft_consensus.cc:686] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986 [term 3 LEADER]: Becoming Leader. State: Replica: f971ee13217e41268e22d92d2a111986, State: Running, Role: LEADER
I20260619 14:34:46.455446 16930 consensus_queue.cc:237] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f971ee13217e41268e22d92d2a111986" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 46797 } }
I20260619 14:34:46.456070 16926 sys_catalog.cc:564] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:34:46.456944 16932 sys_catalog.cc:455] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f971ee13217e41268e22d92d2a111986. Latest consensus state: current_term: 3 leader_uuid: "f971ee13217e41268e22d92d2a111986" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f971ee13217e41268e22d92d2a111986" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 46797 } } }
I20260619 14:34:46.456951 16931 sys_catalog.cc:455] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "f971ee13217e41268e22d92d2a111986" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f971ee13217e41268e22d92d2a111986" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 46797 } } }
I20260619 14:34:46.457091 16932 sys_catalog.cc:458] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986 [sys.catalog]: This master's current role is: LEADER
I20260619 14:34:46.457098 16931 sys_catalog.cc:458] T 00000000000000000000000000000000 P f971ee13217e41268e22d92d2a111986 [sys.catalog]: This master's current role is: LEADER
I20260619 14:34:46.457545 16939 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:34:46.462863 16939 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:34:46.463677 16939 catalog_manager.cc:1259] Loaded cluster ID: f623e242ae6241d69aa547d8601a075c
I20260619 14:34:46.463747 16939 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260619 14:34:46.464808 16939 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f24aee479ad google::LogMessage::Fail() at ??:0
@ 0x7f24aee4a88a google::LogMessage::SendToLog() at ??:0
@ 0x7f24aee474de google::LogMessage::Flush() at ??:0
@ 0x7f24aee48a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f24b9786fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f24b9787d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f24b9784579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f24b97c8640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f24b10993d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f24b1099c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f24b109b3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f24b108a512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f24b26ce6db start_thread at ??:0
@ 0x7f24ad94561f clone at ??:0
W20260619 14:34:46.545553 16928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:34:46.545862 16928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:34:46.545912 16928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:34:46.550709 16928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:34:46.550800 16928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:34:46.550920 16928 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.1
I20260619 14:34:46.556738 16928 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:39547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab
--principal=kudu/127.2.2.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.1:46097
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.2.1
--webserver_port=38943
--tserver_master_addrs=127.2.2.62:46797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16928
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.1
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:34:46.557943 16928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:34:46.559105 16928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:34:46.570006 16967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:34:46.570039 16968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:34:46.570472 16970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:34:46.570456 16928 server_base.cc:1034] running on GCE node
I20260619 14:34:46.570935 16928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:34:46.571579 16928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:34:46.572767 16928 hybrid_clock.cc:648] HybridClock initialized: now 1781879686572736 us; error 51 us; skew 500 ppm
Jun 19 14:34:46 dist-test-slave-70rm krb5kdc[15780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879686, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:34:46.575730 16928 init.cc:377] Logged in from keytab as kudu/127.2.2.1@KRBTEST.COM (short username kudu)
I20260619 14:34:46.576766 16928 webserver.cc:460] Webserver started at http://127.2.2.1:38943/ using document root <none> and password file <none>
I20260619 14:34:46.577358 16928 fs_manager.cc:359] Metadata directory not provided
I20260619 14:34:46.577421 16928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:34:46.581574 16928 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:34:46.583906 16977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:34:46.585353 16928 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 14:34:46.585491 16928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-0/wal
uuid: "c71ce75c24df481097464fb1957b76b4"
format_stamp: "Formatted at 2026-06-19 14:34:44 on dist-test-slave-70rm"
server_key: "a51a61d3af2563badf3e5fbec931e7cc"
server_key_iv: "cdb1c3a55a66061db5e6c614947ba594"
server_key_version: "encryptionkey@0"
I20260619 14:34:46.586179 16928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:34:46.614722 16928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:34:46.618325 16928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:34:46.618543 16928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:34:46.619170 16928 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:34:46.620335 16984 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 14:34:46.625985 16928 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 14:34:46.626101 16928 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260619 14:34:46.626214 16928 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 14:34:46.628003 16928 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 14:34:46.628086 16928 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260619 14:34:46.628414 16984 tablet_bootstrap.cc:492] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4: Bootstrap starting.
I20260619 14:34:46.639196 16928 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.1:46097
I20260619 14:34:46.639472 17091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.1:46097 every 8 connection(s)
I20260619 14:34:46.640277 16928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
I20260619 14:34:46.646450 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 16928
I20260619 14:34:46.646901 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.2:43741
--local_ip_for_outbound_sockets=127.2.2.2
--tserver_master_addrs=127.2.2.62:46797
--webserver_port=44979
--webserver_interface=127.2.2.2
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab
--principal=kudu/127.2.2.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:39547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:34:46.651983 16984 log.cc:824] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4: Log is configured to *not* fsync() on all Append() calls
I20260619 14:34:46.661742 16984 tablet_bootstrap.cc:492] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:34:46.662267 16984 tablet_bootstrap.cc:492] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4: Bootstrap complete.
I20260619 14:34:46.663810 16984 ts_tablet_manager.cc:1397] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4: Time spent bootstrapping tablet: real 0.036s user 0.025s sys 0.008s
I20260619 14:34:46.667385 16984 raft_consensus.cc:348] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a33368758b4a86a88156168ef4c6ec" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 43741 } } peers { permanent_uuid: "6c43506880b44e55b9fdfefbb44f2845" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 34689 } } peers { permanent_uuid: "c71ce75c24df481097464fb1957b76b4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 46097 } }
I20260619 14:34:46.667965 16984 raft_consensus.cc:729] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c71ce75c24df481097464fb1957b76b4, State: Initialized, Role: FOLLOWER
I20260619 14:34:46.668529 16984 consensus_queue.cc:260] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a33368758b4a86a88156168ef4c6ec" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 43741 } } peers { permanent_uuid: "6c43506880b44e55b9fdfefbb44f2845" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 34689 } } peers { permanent_uuid: "c71ce75c24df481097464fb1957b76b4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 46097 } }
I20260619 14:34:46.669682 16984 ts_tablet_manager.cc:1428] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
W20260619 14:34:46.701088 17094 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:34:46.642436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46797 (local address 127.2.2.1:44809)
0619 14:34:46.642925 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:46.642982 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 14:34:46.643934 (+ 952us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:34:46.700120 (+ 56186us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:46797: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":141,"threads_started":1}
W20260619 14:34:46.701730 17092 heartbeater.cc:644] Failed to heartbeat to 127.2.2.62:46797 (0 consecutive failures): Network error: Failed to ping master at 127.2.2.62:46797: Client connection negotiation failed: client connection to 127.2.2.62:46797: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 14:34:46.702622 17094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:46.702351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46797 (local address 127.2.2.1:49645)
0619 14:34:46.702486 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:46.702561 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 14:34:46.703301 17094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:46.703081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46797 (local address 127.2.2.1:34945)
0619 14:34:46.703190 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:46.703248 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 14:34:46.703466 17092 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 14:34:46.766884 17095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:34:46.767179 17095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:34:46.767273 17095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:34:46.771405 17095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:34:46.771498 17095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:34:46.771585 17095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.2
I20260619 14:34:46.777122 17095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:39547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab
--principal=kudu/127.2.2.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.2:43741
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.2.2
--webserver_port=44979
--tserver_master_addrs=127.2.2.62:46797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.2
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:34:46.778280 17095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:34:46.779335 17095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:34:46.789306 17103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:34:46.789456 17095 server_base.cc:1034] running on GCE node
W20260619 14:34:46.789359 17105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:34:46.789354 17102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:34:46.790031 17095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:34:46.790709 17095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:34:46.791903 17095 hybrid_clock.cc:648] HybridClock initialized: now 1781879686791885 us; error 32 us; skew 500 ppm
Jun 19 14:34:46 dist-test-slave-70rm krb5kdc[15780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879686, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:34:46.794924 17095 init.cc:377] Logged in from keytab as kudu/127.2.2.2@KRBTEST.COM (short username kudu)
I20260619 14:34:46.795909 17095 webserver.cc:460] Webserver started at http://127.2.2.2:44979/ using document root <none> and password file <none>
I20260619 14:34:46.796736 17095 fs_manager.cc:359] Metadata directory not provided
I20260619 14:34:46.796849 17095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:34:46.800848 17095 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:34:46.802958 17112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:34:46.804243 17095 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:34:46.804422 17095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-1/wal
uuid: "55a33368758b4a86a88156168ef4c6ec"
format_stamp: "Formatted at 2026-06-19 14:34:44 on dist-test-slave-70rm"
server_key: "809f69815f8eca585e8ea3aca958ee8b"
server_key_iv: "732a1081bc3a6ee3233fb5a1cdb2f226"
server_key_version: "encryptionkey@0"
I20260619 14:34:46.804960 17095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:34:46.828240 17095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:34:46.831754 17095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:34:46.831970 17095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:34:46.832605 17095 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:34:46.833457 17095 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:34:46.833504 17095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:34:46.833585 17095 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:34:46.833618 17095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:34:46.842271 17095 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.2:43741
I20260619 14:34:46.842291 17225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.2:43741 every 8 connection(s)
I20260619 14:34:46.843302 17095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
I20260619 14:34:46.844888 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 17095
I20260619 14:34:46.845285 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.3:34689
--local_ip_for_outbound_sockets=127.2.2.3
--tserver_master_addrs=127.2.2.62:46797
--webserver_port=45009
--webserver_interface=127.2.2.3
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab
--principal=kudu/127.2.2.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:39547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:34:46.847506 17228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:46.845569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46797 (local address 127.2.2.2:52761)
0619 14:34:46.846540 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:46.846679 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":756,"thread_start_us":611,"threads_started":1}
W20260619 14:34:46.848081 17226 heartbeater.cc:644] Failed to heartbeat to 127.2.2.62:46797 (0 consecutive failures): Network error: Failed to ping master at 127.2.2.62:46797: Client connection negotiation failed: client connection to 127.2.2.62:46797: connect: Connection refused (error 111)
I20260619 14:34:46.848709 17228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:46.848486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46797 (local address 127.2.2.2:34535)
0619 14:34:46.848598 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:46.848647 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 14:34:46.849362 17228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:46.849160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46797 (local address 127.2.2.2:51863)
0619 14:34:46.849261 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:46.849312 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 14:34:46.849548 17226 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 14:34:46.958362 17229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:34:46.958683 17229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:34:46.958793 17229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:34:46.963047 17229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:34:46.963135 17229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:34:46.963222 17229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.3
I20260619 14:34:46.968745 17229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:39547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab
--principal=kudu/127.2.2.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.3:34689
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.2.3
--webserver_port=45009
--tserver_master_addrs=127.2.2.62:46797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.3
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:34:46.969861 17229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:34:46.971011 17229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:34:46.980707 17235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:34:46.980715 17237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:34:46.980763 17234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:34:46.980896 17229 server_base.cc:1034] running on GCE node
I20260619 14:34:46.981544 17229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:34:46.982281 17229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:34:46.983492 17229 hybrid_clock.cc:648] HybridClock initialized: now 1781879686983468 us; error 34 us; skew 500 ppm
Jun 19 14:34:46 dist-test-slave-70rm krb5kdc[15780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879686, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:34:46.986645 17229 init.cc:377] Logged in from keytab as kudu/127.2.2.3@KRBTEST.COM (short username kudu)
I20260619 14:34:46.987602 17229 webserver.cc:460] Webserver started at http://127.2.2.3:45009/ using document root <none> and password file <none>
I20260619 14:34:46.988176 17229 fs_manager.cc:359] Metadata directory not provided
I20260619 14:34:46.988270 17229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:34:46.992514 17229 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:34:46.994943 17244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:34:46.996397 17229 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 14:34:46.996548 17229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-2/wal
uuid: "6c43506880b44e55b9fdfefbb44f2845"
format_stamp: "Formatted at 2026-06-19 14:34:44 on dist-test-slave-70rm"
server_key: "82a6f7b11a093d9d44720b057201e669"
server_key_iv: "fb581898a9f147bf76482f2eb9f83942"
server_key_version: "encryptionkey@0"
I20260619 14:34:46.997139 17229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:34:47.023945 17229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:34:47.027303 17229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:34:47.027532 17229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:34:47.028139 17229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:34:47.028978 17229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:34:47.029067 17229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:34:47.029139 17229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:34:47.029174 17229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:34:47.038378 17229 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.3:34689
I20260619 14:34:47.038404 17357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.3:34689 every 8 connection(s)
I20260619 14:34:47.039394 17229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
I20260619 14:34:47.042167 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 17229
I20260619 14:34:47.043363 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:47.041444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46797 (local address 127.2.2.3:52233)
0619 14:34:47.042276 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:47.042456 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":110,"threads_started":1}
W20260619 14:34:47.043787 17358 heartbeater.cc:644] Failed to heartbeat to 127.2.2.62:46797 (0 consecutive failures): Network error: Failed to ping master at 127.2.2.62:46797: Client connection negotiation failed: client connection to 127.2.2.62:46797: connect: Connection refused (error 111)
I20260619 14:34:47.044395 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:47.044176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46797 (local address 127.2.2.3:53573)
0619 14:34:47.044286 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:47.044339 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 14:34:47.045024 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:47.044839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46797 (local address 127.2.2.3:50663)
0619 14:34:47.044936 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:47.044977 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 14:34:47.045228 17358 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 14:34:47.704512 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:47.704086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46797 (local address 127.2.2.1:55503)
0619 14:34:47.704339 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:47.704395 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":73,"threads_started":1}
I20260619 14:34:47.850566 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:47.850111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46797 (local address 127.2.2.2:46379)
0619 14:34:47.850403 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:47.850464 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":80,"threads_started":1}
I20260619 14:34:48.046290 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:48.045813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46797 (local address 127.2.2.3:60135)
0619 14:34:48.046072 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:48.046190 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":72,"threads_started":1}
I20260619 14:34:48.192991 17365 raft_consensus.cc:482] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:34:48.193222 17365 raft_consensus.cc:504] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a33368758b4a86a88156168ef4c6ec" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 43741 } } peers { permanent_uuid: "6c43506880b44e55b9fdfefbb44f2845" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 34689 } } peers { permanent_uuid: "c71ce75c24df481097464fb1957b76b4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 46097 } }
I20260619 14:34:48.194496 17365 leader_election.cc:288] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 55a33368758b4a86a88156168ef4c6ec (127.2.2.2:43741), 6c43506880b44e55b9fdfefbb44f2845 (127.2.2.3:34689)
Jun 19 14:34:48 dist-test-slave-70rm krb5kdc[15780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879686, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.1@KRBTEST.COM for kudu/127.2.2.3@KRBTEST.COM
Jun 19 14:34:48 dist-test-slave-70rm krb5kdc[15780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879686, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.1@KRBTEST.COM for kudu/127.2.2.2@KRBTEST.COM
I20260619 14:34:48.208753 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:48.194916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:57037 (local address 127.2.2.2:43741)
0619 14:34:48.195457 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0619 14:34:48.195482 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0619 14:34:48.195513 (+ 31us) server_negotiation.cc:407] Connection header received
0619 14:34:48.196362 (+ 849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:34:48.196370 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:34:48.196680 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:34:48.196979 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:34:48.198182 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:48.199123 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:48.200342 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:48.200806 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:48.201034 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:48.204050 (+ 3016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:34:48.204054 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:34:48.204065 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:34:48.204094 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:34:48.205875 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:48.206558 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:48.206564 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:48.206570 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:48.206724 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:48.207145 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:48.207152 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:48.207156 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:48.207609 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:34:48.207709 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 14:34:48.208273 (+ 564us) server_negotiation.cc:299] Negotiation successful
0619 14:34:48.208533 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":92,"threads_started":1}
I20260619 14:34:48.208743 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:48.194753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.3:34689 (local address 127.2.2.1:51945)
0619 14:34:48.195097 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:48.195113 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:34:48.195208 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:34:48.197030 (+ 1822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:34:48.197037 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:34:48.197578 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:34:48.197989 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:48.198004 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:48.199266 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:48.199272 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:48.200213 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:48.200222 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:48.200911 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:48.200977 (+ 66us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:48.201171 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:48.201838 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:34:48.201854 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:34:48.203490 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:34:48.205901 (+ 2411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:34:48.205909 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:34:48.205925 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:34:48.206435 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:34:48.207113 (+ 678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:34:48.207118 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:34:48.207120 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:34:48.207213 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:34:48.207821 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:34:48.207824 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:34:48.207963 (+ 139us) client_negotiation.cc:769] Sending connection context
0619 14:34:48.208159 (+ 196us) client_negotiation.cc:240] Negotiation successful
0619 14:34:48.208519 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":152,"spinlock_wait_cycles":30208,"thread_start_us":77,"threads_started":1}
I20260619 14:34:48.208753 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:48.194897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:51945 (local address 127.2.2.3:34689)
0619 14:34:48.195408 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0619 14:34:48.195443 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0619 14:34:48.195489 (+ 46us) server_negotiation.cc:407] Connection header received
0619 14:34:48.196324 (+ 835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:34:48.196331 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:34:48.196680 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:34:48.196987 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:34:48.198181 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:48.199127 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:48.200361 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:34:48.200804 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:34:48.200996 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:48.203753 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:34:48.203762 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:34:48.203781 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:34:48.203820 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:34:48.205771 (+ 1951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:48.206860 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:48.206866 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:48.206870 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:48.207014 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:34:48.207309 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:34:48.207312 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:34:48.207313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:34:48.207706 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:34:48.207848 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 14:34:48.208269 (+ 421us) server_negotiation.cc:299] Negotiation successful
0619 14:34:48.208554 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":90,"threads_started":1}
I20260619 14:34:48.208781 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:48.194699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.2:43741 (local address 127.2.2.1:57037)
0619 14:34:48.194894 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:48.194910 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:34:48.195116 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:34:48.197030 (+ 1914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:34:48.197036 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:34:48.197578 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:34:48.197990 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:48.198004 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:48.199283 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:48.199286 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:48.200212 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:34:48.200222 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:34:48.200945 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:34:48.200963 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:34:48.201159 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:34:48.201902 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:34:48.201914 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:34:48.203874 (+ 1960us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:34:48.205989 (+ 2115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:34:48.205991 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:34:48.205993 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:34:48.206436 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:34:48.206851 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:34:48.206854 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:34:48.206855 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:34:48.207062 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:34:48.207728 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:34:48.207737 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:34:48.207970 (+ 233us) client_negotiation.cc:769] Sending connection context
0619 14:34:48.208158 (+ 188us) client_negotiation.cc:240] Negotiation successful
0619 14:34:48.208608 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":221,"spinlock_wait_cycles":16256}
W20260619 14:34:48.211383 16981 leader_election.cc:341] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6c43506880b44e55b9fdfefbb44f2845 (127.2.2.3:34689): Not found: Tablet not found: b205a6a6b973472097481a8ee9b9e588
W20260619 14:34:48.211556 16980 leader_election.cc:341] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 55a33368758b4a86a88156168ef4c6ec (127.2.2.2:43741): Not found: Tablet not found: b205a6a6b973472097481a8ee9b9e588
I20260619 14:34:48.211632 16981 leader_election.cc:302] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c71ce75c24df481097464fb1957b76b4; no voters: 55a33368758b4a86a88156168ef4c6ec, 6c43506880b44e55b9fdfefbb44f2845
I20260619 14:34:48.211900 17365 raft_consensus.cc:2726] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 14:34:48.705660 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:48.705347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46797 (local address 127.2.2.1:41201)
0619 14:34:48.705500 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:48.705573 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260619 14:34:48.851838 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:48.851394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46797 (local address 127.2.2.2:57713)
0619 14:34:48.851673 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:48.851735 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":86,"threads_started":1}
I20260619 14:34:49.047448 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:49.047063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46797 (local address 127.2.2.3:40425)
0619 14:34:49.047300 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:49.047357 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":73,"threads_started":1}
I20260619 14:34:49.706939 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:49.706524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46797 (local address 127.2.2.1:34281)
0619 14:34:49.706782 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:49.706843 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":87,"threads_started":1}
I20260619 14:34:49.853047 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:49.852645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46797 (local address 127.2.2.2:44899)
0619 14:34:49.852899 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:49.852954 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":94,"threads_started":1}
I20260619 14:34:50.048612 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:50.048179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46797 (local address 127.2.2.3:34567)
0619 14:34:50.048435 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:50.048510 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":87,"threads_started":1}
I20260619 14:34:50.418969 17374 raft_consensus.cc:482] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:34:50.419100 17374 raft_consensus.cc:504] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a33368758b4a86a88156168ef4c6ec" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 43741 } } peers { permanent_uuid: "6c43506880b44e55b9fdfefbb44f2845" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 34689 } } peers { permanent_uuid: "c71ce75c24df481097464fb1957b76b4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 46097 } }
I20260619 14:34:50.419646 17374 leader_election.cc:288] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 55a33368758b4a86a88156168ef4c6ec (127.2.2.2:43741), 6c43506880b44e55b9fdfefbb44f2845 (127.2.2.3:34689)
W20260619 14:34:50.420431 16980 leader_election.cc:341] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 55a33368758b4a86a88156168ef4c6ec (127.2.2.2:43741): Not found: Tablet not found: b205a6a6b973472097481a8ee9b9e588
W20260619 14:34:50.420568 16981 leader_election.cc:341] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6c43506880b44e55b9fdfefbb44f2845 (127.2.2.3:34689): Not found: Tablet not found: b205a6a6b973472097481a8ee9b9e588
I20260619 14:34:50.420666 16981 leader_election.cc:302] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c71ce75c24df481097464fb1957b76b4; no voters: 55a33368758b4a86a88156168ef4c6ec, 6c43506880b44e55b9fdfefbb44f2845
I20260619 14:34:50.420866 17374 raft_consensus.cc:2726] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 14:34:50.708125 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:50.707670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46797 (local address 127.2.2.1:56145)
0619 14:34:50.707971 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:50.708027 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":124,"threads_started":1}
I20260619 14:34:50.854269 17376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:50.853803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46797 (local address 127.2.2.2:49679)
0619 14:34:50.854047 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:50.854105 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":73,"threads_started":1}
I20260619 14:34:51.049861 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:51.049392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46797 (local address 127.2.2.3:39577)
0619 14:34:51.049691 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:51.049752 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":85,"threads_started":1}
I20260619 14:34:51.709445 17378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:51.709019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46797 (local address 127.2.2.1:35047)
0619 14:34:51.709286 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:51.709351 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":67,"threads_started":1}
I20260619 14:34:51.855485 17379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:51.855074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46797 (local address 127.2.2.2:57961)
0619 14:34:51.855325 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:51.855388 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":73,"threads_started":1}
I20260619 14:34:52.051232 17380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:52.050824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46797 (local address 127.2.2.3:42167)
0619 14:34:52.051080 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:52.051140 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":80,"threads_started":1}
I20260619 14:34:52.710788 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:52.710331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46797 (local address 127.2.2.1:53081)
0619 14:34:52.710623 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:52.710689 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":79,"threads_started":1}
I20260619 14:34:52.856588 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:52.856222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46797 (local address 127.2.2.2:38953)
0619 14:34:52.856448 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:52.856500 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":70,"threads_started":1}
I20260619 14:34:53.052594 17383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:53.052098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46797 (local address 127.2.2.3:32987)
0619 14:34:53.052387 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:53.052454 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":81,"threads_started":1}
I20260619 14:34:53.072347 17384 raft_consensus.cc:482] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:34:53.072490 17384 raft_consensus.cc:504] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a33368758b4a86a88156168ef4c6ec" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 43741 } } peers { permanent_uuid: "6c43506880b44e55b9fdfefbb44f2845" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 34689 } } peers { permanent_uuid: "c71ce75c24df481097464fb1957b76b4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 46097 } }
I20260619 14:34:53.072988 17384 leader_election.cc:288] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 55a33368758b4a86a88156168ef4c6ec (127.2.2.2:43741), 6c43506880b44e55b9fdfefbb44f2845 (127.2.2.3:34689)
W20260619 14:34:53.073938 16981 leader_election.cc:341] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6c43506880b44e55b9fdfefbb44f2845 (127.2.2.3:34689): Not found: Tablet not found: b205a6a6b973472097481a8ee9b9e588
W20260619 14:34:53.074072 16980 leader_election.cc:341] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 55a33368758b4a86a88156168ef4c6ec (127.2.2.2:43741): Not found: Tablet not found: b205a6a6b973472097481a8ee9b9e588
I20260619 14:34:53.074195 16980 leader_election.cc:302] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c71ce75c24df481097464fb1957b76b4; no voters: 55a33368758b4a86a88156168ef4c6ec, 6c43506880b44e55b9fdfefbb44f2845
I20260619 14:34:53.074357 17384 raft_consensus.cc:2726] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 14:34:53.712067 17385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:53.711593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46797 (local address 127.2.2.1:50065)
0619 14:34:53.711858 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:53.711912 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":96,"threads_started":1}
I20260619 14:34:53.857874 17386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:53.857407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46797 (local address 127.2.2.2:33843)
0619 14:34:53.857713 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:53.857771 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":95,"threads_started":1}
I20260619 14:34:54.053831 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:54.053351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46797 (local address 127.2.2.3:59891)
0619 14:34:54.053676 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:54.053735 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":82,"threads_started":1}
I20260619 14:34:54.713260 17388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:54.712810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46797 (local address 127.2.2.1:38371)
0619 14:34:54.713093 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:54.713159 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":73,"threads_started":1}
I20260619 14:34:54.859040 17389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:54.858632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46797 (local address 127.2.2.2:59085)
0619 14:34:54.858885 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:54.858947 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":73,"threads_started":1}
I20260619 14:34:55.055130 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:55.054666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46797 (local address 127.2.2.3:32871)
0619 14:34:55.054952 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:55.055028 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":72,"threads_started":1}
I20260619 14:34:55.714572 17391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:55.714076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46797 (local address 127.2.2.1:38861)
0619 14:34:55.714411 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:55.714475 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":134,"threads_started":1}
I20260619 14:34:55.860376 17392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:55.859866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46797 (local address 127.2.2.2:39869)
0619 14:34:55.860137 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:55.860219 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":98,"threads_started":1}
I20260619 14:34:56.056424 17393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:56.055997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46797 (local address 127.2.2.3:51033)
0619 14:34:56.056251 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:56.056330 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":76,"threads_started":1}
I20260619 14:34:56.080127 17394 raft_consensus.cc:482] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:34:56.080252 17394 raft_consensus.cc:504] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a33368758b4a86a88156168ef4c6ec" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 43741 } } peers { permanent_uuid: "6c43506880b44e55b9fdfefbb44f2845" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 34689 } } peers { permanent_uuid: "c71ce75c24df481097464fb1957b76b4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 46097 } }
I20260619 14:34:56.080727 17394 leader_election.cc:288] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 55a33368758b4a86a88156168ef4c6ec (127.2.2.2:43741), 6c43506880b44e55b9fdfefbb44f2845 (127.2.2.3:34689)
W20260619 14:34:56.081560 16981 leader_election.cc:341] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6c43506880b44e55b9fdfefbb44f2845 (127.2.2.3:34689): Not found: Tablet not found: b205a6a6b973472097481a8ee9b9e588
W20260619 14:34:56.081672 16980 leader_election.cc:341] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 55a33368758b4a86a88156168ef4c6ec (127.2.2.2:43741): Not found: Tablet not found: b205a6a6b973472097481a8ee9b9e588
I20260619 14:34:56.081739 16980 leader_election.cc:302] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c71ce75c24df481097464fb1957b76b4; no voters: 55a33368758b4a86a88156168ef4c6ec, 6c43506880b44e55b9fdfefbb44f2845
I20260619 14:34:56.081879 17394 raft_consensus.cc:2726] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 14:34:56.715914 17395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:56.715478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46797 (local address 127.2.2.1:55257)
0619 14:34:56.715754 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:56.715817 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":104,"threads_started":1}
I20260619 14:34:56.861706 17396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:56.861306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46797 (local address 127.2.2.2:47517)
0619 14:34:56.861554 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:56.861612 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":69,"threads_started":1}
I20260619 14:34:57.057607 17397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:57.057168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46797 (local address 127.2.2.3:34433)
0619 14:34:57.057458 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:57.057514 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":88,"threads_started":1}
I20260619 14:34:57.717150 17398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:57.716740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46797 (local address 127.2.2.1:53273)
0619 14:34:57.717001 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:57.717053 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":72,"threads_started":1}
I20260619 14:34:57.863039 17399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:57.862580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46797 (local address 127.2.2.2:53305)
0619 14:34:57.862866 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:57.862934 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":76,"threads_started":1}
I20260619 14:34:58.058929 17400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:58.058505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46797 (local address 127.2.2.3:56071)
0619 14:34:58.058776 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:58.058836 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":101,"threads_started":1}
I20260619 14:34:58.718453 17401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:58.717987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46797 (local address 127.2.2.1:52075)
0619 14:34:58.718294 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:58.718354 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":76,"threads_started":1}
I20260619 14:34:58.864454 17402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:58.863993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46797 (local address 127.2.2.2:41751)
0619 14:34:58.864275 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:58.864335 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":105,"threads_started":1}
I20260619 14:34:59.060213 17403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:59.059773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46797 (local address 127.2.2.3:45959)
0619 14:34:59.060053 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:59.060118 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":81,"threads_started":1}
I20260619 14:34:59.719666 17404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:59.719256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46797 (local address 127.2.2.1:33857)
0619 14:34:59.719514 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:59.719570 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":93,"threads_started":1}
I20260619 14:34:59.865664 17405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:34:59.865232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46797 (local address 127.2.2.2:50773)
0619 14:34:59.865511 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0619 14:34:59.865567 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":95,"threads_started":1}
I20260619 14:35:00.061367 17406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:00.060967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46797 (local address 127.2.2.3:39941)
0619 14:35:00.061220 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:00.061275 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":90,"threads_started":1}
I20260619 14:35:00.720841 17407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:00.720401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46797 (local address 127.2.2.1:49219)
0619 14:35:00.720670 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:00.720744 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":79,"threads_started":1}
I20260619 14:35:00.866966 17408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:00.866527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46797 (local address 127.2.2.2:41121)
0619 14:35:00.866804 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:00.866867 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":77,"threads_started":1}
I20260619 14:35:01.062604 17409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:01.062190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46797 (local address 127.2.2.3:54235)
0619 14:35:01.062451 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:01.062511 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":91,"threads_started":1}
I20260619 14:35:01.396343 17410 raft_consensus.cc:482] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:35:01.396543 17410 raft_consensus.cc:504] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a33368758b4a86a88156168ef4c6ec" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 43741 } } peers { permanent_uuid: "6c43506880b44e55b9fdfefbb44f2845" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 34689 } } peers { permanent_uuid: "c71ce75c24df481097464fb1957b76b4" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 46097 } }
I20260619 14:35:01.397012 17410 leader_election.cc:288] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 55a33368758b4a86a88156168ef4c6ec (127.2.2.2:43741), 6c43506880b44e55b9fdfefbb44f2845 (127.2.2.3:34689)
W20260619 14:35:01.397768 16980 leader_election.cc:341] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 55a33368758b4a86a88156168ef4c6ec (127.2.2.2:43741): Not found: Tablet not found: b205a6a6b973472097481a8ee9b9e588
W20260619 14:35:01.397866 16981 leader_election.cc:341] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6c43506880b44e55b9fdfefbb44f2845 (127.2.2.3:34689): Not found: Tablet not found: b205a6a6b973472097481a8ee9b9e588
I20260619 14:35:01.397987 16981 leader_election.cc:302] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c71ce75c24df481097464fb1957b76b4; no voters: 55a33368758b4a86a88156168ef4c6ec, 6c43506880b44e55b9fdfefbb44f2845
I20260619 14:35:01.398187 17410 raft_consensus.cc:2726] T b205a6a6b973472097481a8ee9b9e588 P c71ce75c24df481097464fb1957b76b4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 14:35:01.722295 17411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:01.721837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46797 (local address 127.2.2.1:39263)
0619 14:35:01.722089 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:01.722199 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":82,"threads_started":1}
I20260619 14:35:01.868404 17412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:01.868008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46797 (local address 127.2.2.2:35579)
0619 14:35:01.868252 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:01.868307 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":78,"threads_started":1}
I20260619 14:35:02.043354 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 16928
I20260619 14:35:02.050364 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 17095
I20260619 14:35:02.056079 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 17229
2026-06-19T14:35:02Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19501 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:35:02 dist-test-slave-70rm krb5kdc[17415](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:35:02 dist-test-slave-70rm krb5kdc[17415](info): set up 2 sockets
Jun 19 14:35:02 dist-test-slave-70rm krb5kdc[17415](info): commencing operation
krb5kdc: starting...
W20260619 14:35:03.098489 2056 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Jun 19 14:35:03 dist-test-slave-70rm krb5kdc[17415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:35:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:35:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.2.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab.
Entry for principal kudu/127.2.2.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab.
I20260619 14:35:03.224880 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:41891
--webserver_interface=127.2.2.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:37921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.2.62:41891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:35:03.335400 17428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:35:03.335733 17428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:35:03.335805 17428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:35:03.339973 17428 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 14:35:03.340059 17428 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 14:35:03.340087 17428 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 14:35:03.340118 17428 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:35:03.340159 17428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:35:03.340202 17428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:35:03.340253 17428 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:35:03.340293 17428 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:35:03.345615 17428 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:37921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.2.62:41891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:41891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.2.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:35:03.346761 17428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:35:03.347772 17428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:35:03.356441 17433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:35:03.356441 17434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:35:03.356714 17428 server_base.cc:1034] running on GCE node
W20260619 14:35:03.356442 17436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:35:03.357388 17428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:35:03.358582 17428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:35:03.359743 17428 hybrid_clock.cc:648] HybridClock initialized: now 1781879703359720 us; error 38 us; skew 500 ppm
Jun 19 14:35:03 dist-test-slave-70rm krb5kdc[17415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879703, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:35:03.362645 17428 init.cc:377] Logged in from keytab as kudu/127.2.2.62@KRBTEST.COM (short username kudu)
I20260619 14:35:03.363723 17428 webserver.cc:460] Webserver started at http://127.2.2.62:34103/ using document root <none> and password file <none>
I20260619 14:35:03.364295 17428 fs_manager.cc:359] Metadata directory not provided
I20260619 14:35:03.364377 17428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:35:03.364605 17428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:35:03.367090 17428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/master-0/data/instance:
uuid: "b57bdd13c44f4b00a3d67fc5b168cb3f"
format_stamp: "Formatted at 2026-06-19 14:35:03 on dist-test-slave-70rm"
server_key: "536d4638e100092d76e2479fb40d531e"
server_key_iv: "b9de7b5fc7fe0ee9164acb5b1e8cea70"
server_key_version: "encryptionkey@0"
I20260619 14:35:03.367658 17428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/master-0/wal/instance:
uuid: "b57bdd13c44f4b00a3d67fc5b168cb3f"
format_stamp: "Formatted at 2026-06-19 14:35:03 on dist-test-slave-70rm"
server_key: "536d4638e100092d76e2479fb40d531e"
server_key_iv: "b9de7b5fc7fe0ee9164acb5b1e8cea70"
server_key_version: "encryptionkey@0"
I20260619 14:35:03.371415 17428 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:35:03.373873 17443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:35:03.375146 17428 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:35:03.375319 17428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/master-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/master-0/wal
uuid: "b57bdd13c44f4b00a3d67fc5b168cb3f"
format_stamp: "Formatted at 2026-06-19 14:35:03 on dist-test-slave-70rm"
server_key: "536d4638e100092d76e2479fb40d531e"
server_key_iv: "b9de7b5fc7fe0ee9164acb5b1e8cea70"
server_key_version: "encryptionkey@0"
I20260619 14:35:03.375439 17428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:35:03.389796 17428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:35:03.392946 17428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:35:03.393186 17428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:35:03.400238 17428 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.62:41891
I20260619 14:35:03.400527 17495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.62:41891 every 8 connection(s)
I20260619 14:35:03.401304 17428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
I20260619 14:35:03.404119 17496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:35:03.409817 17496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f: Bootstrap starting.
I20260619 14:35:03.411258 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 17428
I20260619 14:35:03.411381 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/master-0/wal/instance
I20260619 14:35:03.411666 2056 external_mini_cluster.cc:1428] Setting key 79476c12cb2a23075cc86db59e277934
I20260619 14:35:03.412806 17496 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f: Neither blocks nor log segments found. Creating new log.
I20260619 14:35:03.413610 17496 log.cc:824] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f: Log is configured to *not* fsync() on all Append() calls
I20260619 14:35:03.416018 17496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f: No bootstrap required, opened a new log
I20260619 14:35:03.420420 17496 raft_consensus.cc:348] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57bdd13c44f4b00a3d67fc5b168cb3f" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41891 } }
I20260619 14:35:03.420739 17496 raft_consensus.cc:374] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:35:03.420856 17496 raft_consensus.cc:729] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b57bdd13c44f4b00a3d67fc5b168cb3f, State: Initialized, Role: FOLLOWER
I20260619 14:35:03.421322 17496 consensus_queue.cc:260] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57bdd13c44f4b00a3d67fc5b168cb3f" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41891 } }
Jun 19 14:35:03 dist-test-slave-70rm krb5kdc[17415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:35:03.421479 17496 raft_consensus.cc:388] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:35:03.421559 17496 raft_consensus.cc:482] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:35:03.421666 17496 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:35:03.422896 17496 raft_consensus.cc:504] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57bdd13c44f4b00a3d67fc5b168cb3f" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41891 } }
I20260619 14:35:03.423322 17496 leader_election.cc:302] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b57bdd13c44f4b00a3d67fc5b168cb3f; no voters:
I20260619 14:35:03.423717 17496 leader_election.cc:288] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:35:03.423795 17501 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:35:03.423990 17501 raft_consensus.cc:686] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f [term 1 LEADER]: Becoming Leader. State: Replica: b57bdd13c44f4b00a3d67fc5b168cb3f, State: Running, Role: LEADER
I20260619 14:35:03.424362 17501 consensus_queue.cc:237] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57bdd13c44f4b00a3d67fc5b168cb3f" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41891 } }
I20260619 14:35:03.425083 17496 sys_catalog.cc:564] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:35:03.425984 17503 sys_catalog.cc:455] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b57bdd13c44f4b00a3d67fc5b168cb3f. Latest consensus state: current_term: 1 leader_uuid: "b57bdd13c44f4b00a3d67fc5b168cb3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57bdd13c44f4b00a3d67fc5b168cb3f" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41891 } } }
I20260619 14:35:03.426465 17503 sys_catalog.cc:458] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f [sys.catalog]: This master's current role is: LEADER
I20260619 14:35:03.426343 17502 sys_catalog.cc:455] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b57bdd13c44f4b00a3d67fc5b168cb3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57bdd13c44f4b00a3d67fc5b168cb3f" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41891 } } }
I20260619 14:35:03.426843 17502 sys_catalog.cc:458] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f [sys.catalog]: This master's current role is: LEADER
I20260619 14:35:03.427922 17499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:03.412912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53436 (local address 127.2.2.62:41891)
0619 14:35:03.413379 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0619 14:35:03.413397 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 14:35:03.413438 (+ 41us) server_negotiation.cc:407] Connection header received
0619 14:35:03.414347 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:35:03.414374 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:35:03.414794 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:35:03.415104 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:35:03.416046 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:35:03.417121 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:35:03.418340 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:35:03.418866 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:35:03.419006 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:35:03.422112 (+ 3106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:35:03.422117 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:35:03.422173 (+ 56us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:35:03.422209 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:35:03.424462 (+ 2253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:35:03.424988 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:35:03.424994 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:35:03.425000 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:35:03.425147 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:35:03.425427 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:35:03.425430 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:35:03.425432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:35:03.426051 (+ 619us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:35:03.426295 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0619 14:35:03.426582 (+ 287us) server_negotiation.cc:299] Negotiation successful
0619 14:35:03.426926 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":144,"threads_started":1}
W20260619 14:35:03.431118 17516 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:35:03.431207 17516 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:35:03.431283 17511 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:35:03.432022 17511 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:35:03.436993 17511 catalog_manager.cc:1347] Generated new cluster ID: f112e01fc9464ef8b0580b565920bf8a
I20260619 14:35:03.437060 17511 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:35:03.447994 17511 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:35:03.449322 17511 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:35:03.456435 17511 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f: Generated new TSK 0
I20260619 14:35:03.457195 17511 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.2.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab.
Entry for principal kudu/127.2.2.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab.
I20260619 14:35:03.494634 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.1:0
--local_ip_for_outbound_sockets=127.2.2.1
--webserver_interface=127.2.2.1
--webserver_port=0
--tserver_master_addrs=127.2.2.62:41891
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab
--principal=kudu/127.2.2.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:37921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:35:03.619141 17522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:35:03.619448 17522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:35:03.619495 17522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:35:03.623544 17522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:35:03.623615 17522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:35:03.623677 17522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.1
I20260619 14:35:03.629927 17522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:37921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab
--principal=kudu/127.2.2.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.2.1
--webserver_port=0
--tserver_master_addrs=127.2.2.62:41891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17522
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.1
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:35:03.631106 17522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:35:03.632108 17522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:35:03.642087 17527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:35:03.642096 17530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:35:03.642177 17522 server_base.cc:1034] running on GCE node
W20260619 14:35:03.642098 17528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:35:03.642736 17522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:35:03.643425 17522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:35:03.644624 17522 hybrid_clock.cc:648] HybridClock initialized: now 1781879703644603 us; error 36 us; skew 500 ppm
Jun 19 14:35:03 dist-test-slave-70rm krb5kdc[17415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879703, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:35:03.647661 17522 init.cc:377] Logged in from keytab as kudu/127.2.2.1@KRBTEST.COM (short username kudu)
I20260619 14:35:03.648586 17522 webserver.cc:460] Webserver started at http://127.2.2.1:43115/ using document root <none> and password file <none>
I20260619 14:35:03.649165 17522 fs_manager.cc:359] Metadata directory not provided
I20260619 14:35:03.649255 17522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:35:03.649501 17522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:35:03.651648 17522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-0/data/instance:
uuid: "ed48fbe415dd43babd6582e39e83e02c"
format_stamp: "Formatted at 2026-06-19 14:35:03 on dist-test-slave-70rm"
server_key: "063a84231a70ecfe86ea55c3b4623636"
server_key_iv: "f52d9d4620e8c4aacadf7d13af1f6ed4"
server_key_version: "encryptionkey@0"
I20260619 14:35:03.652231 17522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-0/wal/instance:
uuid: "ed48fbe415dd43babd6582e39e83e02c"
format_stamp: "Formatted at 2026-06-19 14:35:03 on dist-test-slave-70rm"
server_key: "063a84231a70ecfe86ea55c3b4623636"
server_key_iv: "f52d9d4620e8c4aacadf7d13af1f6ed4"
server_key_version: "encryptionkey@0"
I20260619 14:35:03.656497 17522 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:35:03.659439 17537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:35:03.660894 17522 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:35:03.661041 17522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-0/wal
uuid: "ed48fbe415dd43babd6582e39e83e02c"
format_stamp: "Formatted at 2026-06-19 14:35:03 on dist-test-slave-70rm"
server_key: "063a84231a70ecfe86ea55c3b4623636"
server_key_iv: "f52d9d4620e8c4aacadf7d13af1f6ed4"
server_key_version: "encryptionkey@0"
I20260619 14:35:03.661177 17522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:35:03.680919 17522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:35:03.684502 17522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:35:03.684717 17522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:35:03.685366 17522 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:35:03.686303 17522 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:35:03.686352 17522 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:35:03.686416 17522 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:35:03.686447 17522 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:35:03.695791 17650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.1:36581 every 8 connection(s)
I20260619 14:35:03.695789 17522 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.1:36581
I20260619 14:35:03.696832 17522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
I20260619 14:35:03.706218 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 17522
I20260619 14:35:03.706358 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-0/wal/instance
I20260619 14:35:03.706707 2056 external_mini_cluster.cc:1428] Setting key 2c10ae09305ac6d4acc07fe99e481c1c
Jun 19 14:35:03 dist-test-slave-70rm krb5kdc[17415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879703, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.1@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:35:03.711580 17499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:03.698820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:52351 (local address 127.2.2.62:41891)
0619 14:35:03.698956 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 14:35:03.698960 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:35:03.699783 (+ 823us) server_negotiation.cc:407] Connection header received
0619 14:35:03.701045 (+ 1262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:35:03.701048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:35:03.701096 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:35:03.701173 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:35:03.702482 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:35:03.703036 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:35:03.704251 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:35:03.704482 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:35:03.704569 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:35:03.707506 (+ 2937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:35:03.707509 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:35:03.707511 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:35:03.707536 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:35:03.709104 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:35:03.709779 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:35:03.709784 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:35:03.709786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:35:03.709911 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:35:03.710309 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:35:03.710314 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:35:03.710316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:35:03.710569 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:35:03.710676 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 14:35:03.711322 (+ 646us) server_negotiation.cc:299] Negotiation successful
0619 14:35:03.711452 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 14:35:03.712863 17653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:03.699172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41891 (local address 127.2.2.1:52351)
0619 14:35:03.699626 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:03.699671 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 14:35:03.700844 (+ 1173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:35:03.701370 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:35:03.701377 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:35:03.701760 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:35:03.702334 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:35:03.702351 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:35:03.703174 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:35:03.703177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:35:03.704130 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:35:03.704140 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:35:03.704571 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:35:03.704622 (+ 51us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:35:03.704793 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:35:03.705407 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:35:03.705424 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:35:03.707351 (+ 1927us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:35:03.709227 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:35:03.709246 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:35:03.709268 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:35:03.709672 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:35:03.710010 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:35:03.710012 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:35:03.710014 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:35:03.710217 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:35:03.710705 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:35:03.710710 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:35:03.710972 (+ 262us) client_negotiation.cc:769] Sending connection context
0619 14:35:03.711207 (+ 235us) client_negotiation.cc:240] Negotiation successful
0619 14:35:03.711612 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":129,"threads_started":1}
I20260619 14:35:03.714413 17651 heartbeater.cc:347] Connected to a master server at 127.2.2.62:41891
I20260619 14:35:03.714680 17651 heartbeater.cc:464] Registering TS with master...
I20260619 14:35:03.715207 17651 heartbeater.cc:511] Master 127.2.2.62:41891 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.2.2@KRBTEST.COM; defaulting to no policy
I20260619 14:35:03.716998 17460 ts_manager.cc:194] Registered new tserver with Master: ed48fbe415dd43babd6582e39e83e02c (127.2.2.1:36581)
I20260619 14:35:03.718286 17460 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.2.1@KRBTEST.COM'} at 127.2.2.1:52351
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab.
Entry for principal kudu/127.2.2.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab.
I20260619 14:35:03.737355 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.2:0
--local_ip_for_outbound_sockets=127.2.2.2
--webserver_interface=127.2.2.2
--webserver_port=0
--tserver_master_addrs=127.2.2.62:41891
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab
--principal=kudu/127.2.2.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:37921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:35:03.849915 17656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:35:03.850255 17656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:35:03.850329 17656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:35:03.854452 17656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:35:03.854512 17656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:35:03.854614 17656 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.2
I20260619 14:35:03.860438 17656 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:37921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab
--principal=kudu/127.2.2.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.2.2
--webserver_port=0
--tserver_master_addrs=127.2.2.62:41891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.2
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:35:03.861560 17656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:35:03.862627 17656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:35:03.872340 17664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:35:03.872416 17656 server_base.cc:1034] running on GCE node
W20260619 14:35:03.872359 17661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:35:03.872357 17662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:35:03.873086 17656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:35:03.873798 17656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:35:03.874950 17656 hybrid_clock.cc:648] HybridClock initialized: now 1781879703874934 us; error 32 us; skew 500 ppm
Jun 19 14:35:03 dist-test-slave-70rm krb5kdc[17415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879703, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:35:03.877653 17656 init.cc:377] Logged in from keytab as kudu/127.2.2.2@KRBTEST.COM (short username kudu)
I20260619 14:35:03.878661 17656 webserver.cc:460] Webserver started at http://127.2.2.2:40855/ using document root <none> and password file <none>
I20260619 14:35:03.879282 17656 fs_manager.cc:359] Metadata directory not provided
I20260619 14:35:03.879377 17656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:35:03.879596 17656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:35:03.881654 17656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-1/data/instance:
uuid: "37ea989b097c44838b4491edae66ac67"
format_stamp: "Formatted at 2026-06-19 14:35:03 on dist-test-slave-70rm"
server_key: "821e7ea015dc5edf0313fdc50aa93224"
server_key_iv: "80baa16d283bb6b366df3cf486b93e40"
server_key_version: "encryptionkey@0"
I20260619 14:35:03.882304 17656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-1/wal/instance:
uuid: "37ea989b097c44838b4491edae66ac67"
format_stamp: "Formatted at 2026-06-19 14:35:03 on dist-test-slave-70rm"
server_key: "821e7ea015dc5edf0313fdc50aa93224"
server_key_iv: "80baa16d283bb6b366df3cf486b93e40"
server_key_version: "encryptionkey@0"
I20260619 14:35:03.886101 17656 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 14:35:03.888559 17671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:35:03.889798 17656 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:35:03.889935 17656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-1/wal
uuid: "37ea989b097c44838b4491edae66ac67"
format_stamp: "Formatted at 2026-06-19 14:35:03 on dist-test-slave-70rm"
server_key: "821e7ea015dc5edf0313fdc50aa93224"
server_key_iv: "80baa16d283bb6b366df3cf486b93e40"
server_key_version: "encryptionkey@0"
I20260619 14:35:03.890048 17656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:35:03.926402 17656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:35:03.929982 17656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:35:03.930220 17656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:35:03.930832 17656 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:35:03.931669 17656 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:35:03.931737 17656 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:35:03.931797 17656 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:35:03.931843 17656 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:35:03.940326 17656 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.2:33101
I20260619 14:35:03.940356 17784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.2:33101 every 8 connection(s)
I20260619 14:35:03.941231 17656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
I20260619 14:35:03.944344 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 17656
I20260619 14:35:03.944458 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-1/wal/instance
I20260619 14:35:03.944810 2056 external_mini_cluster.cc:1428] Setting key a834548a3ff674f52939d7ef2083180e
Jun 19 14:35:03 dist-test-slave-70rm krb5kdc[17415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879703, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.2@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.2.3@KRBTEST.COM; defaulting to no policy
I20260619 14:35:03.957423 17499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:03.943197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:44713 (local address 127.2.2.62:41891)
0619 14:35:03.943336 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 14:35:03.943339 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:35:03.944253 (+ 914us) server_negotiation.cc:407] Connection header received
0619 14:35:03.945881 (+ 1628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:35:03.945885 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:35:03.945932 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:35:03.946019 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:35:03.947676 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:35:03.948400 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:35:03.949585 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:35:03.950004 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:35:03.950153 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:35:03.953061 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:35:03.953067 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:35:03.953073 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:35:03.953109 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:35:03.954988 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:35:03.955619 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:35:03.955622 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:35:03.955623 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:35:03.955705 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:35:03.956083 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:35:03.956088 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:35:03.956090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:35:03.956406 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:35:03.956525 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 14:35:03.957071 (+ 546us) server_negotiation.cc:299] Negotiation successful
0619 14:35:03.957234 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.3@KRBTEST.COM" created.
I20260619 14:35:03.958421 17787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:03.943595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41891 (local address 127.2.2.2:44713)
0619 14:35:03.944039 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:03.944118 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0619 14:35:03.945654 (+ 1536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:35:03.946267 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:35:03.946274 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:35:03.946801 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:35:03.947505 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:35:03.947527 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:35:03.948525 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:35:03.948528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:35:03.949467 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:35:03.949477 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:35:03.950188 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:35:03.950243 (+ 55us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:35:03.950398 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:35:03.951090 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:35:03.951107 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:35:03.952876 (+ 1769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:35:03.955159 (+ 2283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:35:03.955168 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:35:03.955190 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:35:03.955501 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:35:03.955806 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:35:03.955808 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:35:03.955810 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:35:03.955960 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:35:03.956517 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:35:03.956523 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:35:03.956799 (+ 276us) client_negotiation.cc:769] Sending connection context
0619 14:35:03.956998 (+ 199us) client_negotiation.cc:240] Negotiation successful
0619 14:35:03.957367 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":121,"threads_started":1}
I20260619 14:35:03.959612 17785 heartbeater.cc:347] Connected to a master server at 127.2.2.62:41891
I20260619 14:35:03.959853 17785 heartbeater.cc:464] Registering TS with master...
I20260619 14:35:03.960361 17785 heartbeater.cc:511] Master 127.2.2.62:41891 requested a full tablet report, sending...
I20260619 14:35:03.961452 17460 ts_manager.cc:194] Registered new tserver with Master: 37ea989b097c44838b4491edae66ac67 (127.2.2.2:33101)
I20260619 14:35:03.962347 17460 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.2.2@KRBTEST.COM'} at 127.2.2.2:44713
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab.
Entry for principal kudu/127.2.2.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab.
I20260619 14:35:03.974999 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.3:0
--local_ip_for_outbound_sockets=127.2.2.3
--webserver_interface=127.2.2.3
--webserver_port=0
--tserver_master_addrs=127.2.2.62:41891
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab
--principal=kudu/127.2.2.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:37921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:35:04.084277 17790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:35:04.084553 17790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:35:04.084597 17790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:35:04.088500 17790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:35:04.088560 17790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:35:04.088620 17790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.3
I20260619 14:35:04.093917 17790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:37921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab
--principal=kudu/127.2.2.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.2.3
--webserver_port=0
--tserver_master_addrs=127.2.2.62:41891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.3
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:35:04.095119 17790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:35:04.096145 17790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:35:04.105536 17795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:35:04.105551 17798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:35:04.105758 17790 server_base.cc:1034] running on GCE node
W20260619 14:35:04.105541 17796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:35:04.106361 17790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:35:04.107008 17790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:35:04.108197 17790 hybrid_clock.cc:648] HybridClock initialized: now 1781879704108163 us; error 44 us; skew 500 ppm
Jun 19 14:35:04 dist-test-slave-70rm krb5kdc[17415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879704, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:35:04.111104 17790 init.cc:377] Logged in from keytab as kudu/127.2.2.3@KRBTEST.COM (short username kudu)
I20260619 14:35:04.112013 17790 webserver.cc:460] Webserver started at http://127.2.2.3:44157/ using document root <none> and password file <none>
I20260619 14:35:04.112591 17790 fs_manager.cc:359] Metadata directory not provided
I20260619 14:35:04.112681 17790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:35:04.112919 17790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:35:04.115082 17790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-2/data/instance:
uuid: "8d1b5a5b42be462895944884b9163686"
format_stamp: "Formatted at 2026-06-19 14:35:04 on dist-test-slave-70rm"
server_key: "01fabbf4a4bacd1958c69260620e5990"
server_key_iv: "255e76cd5e1a3ae84d6154042addcaa4"
server_key_version: "encryptionkey@0"
I20260619 14:35:04.115651 17790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-2/wal/instance:
uuid: "8d1b5a5b42be462895944884b9163686"
format_stamp: "Formatted at 2026-06-19 14:35:04 on dist-test-slave-70rm"
server_key: "01fabbf4a4bacd1958c69260620e5990"
server_key_iv: "255e76cd5e1a3ae84d6154042addcaa4"
server_key_version: "encryptionkey@0"
I20260619 14:35:04.119401 17790 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:35:04.121949 17805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:35:04.123274 17790 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:35:04.123428 17790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-2/wal
uuid: "8d1b5a5b42be462895944884b9163686"
format_stamp: "Formatted at 2026-06-19 14:35:04 on dist-test-slave-70rm"
server_key: "01fabbf4a4bacd1958c69260620e5990"
server_key_iv: "255e76cd5e1a3ae84d6154042addcaa4"
server_key_version: "encryptionkey@0"
I20260619 14:35:04.123554 17790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:35:04.147825 17790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:35:04.151276 17790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:35:04.151479 17790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:35:04.152120 17790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:35:04.152966 17790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:35:04.153033 17790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:35:04.153100 17790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:35:04.153146 17790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:35:04.162173 17790 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.3:45385
I20260619 14:35:04.162207 17918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.3:45385 every 8 connection(s)
I20260619 14:35:04.163115 17790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
Jun 19 14:35:04 dist-test-slave-70rm krb5kdc[17415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879704, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.3@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:35:04.172600 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 17790
I20260619 14:35:04.172729 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-2/wal/instance
I20260619 14:35:04.173040 2056 external_mini_cluster.cc:1428] Setting key 2bd091de8e90e73372ecb84a482473ba
I20260619 14:35:04.177042 17499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:04.164911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.3:49545 (local address 127.2.2.62:41891)
0619 14:35:04.165043 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 14:35:04.165047 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:35:04.165883 (+ 836us) server_negotiation.cc:407] Connection header received
0619 14:35:04.167217 (+ 1334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:35:04.167220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:35:04.167271 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:35:04.167346 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:35:04.168476 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:35:04.169013 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:35:04.170201 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:35:04.170426 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:35:04.170509 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:35:04.173091 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:35:04.173095 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:35:04.173098 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:35:04.173130 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:35:04.174864 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:35:04.175513 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:35:04.175516 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:35:04.175517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:35:04.175599 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:35:04.175926 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:35:04.175929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:35:04.175930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:35:04.176177 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:35:04.176275 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 14:35:04.176804 (+ 529us) server_negotiation.cc:299] Negotiation successful
0619 14:35:04.176924 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 14:35:04.177934 17921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:04.165259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41891 (local address 127.2.2.3:49545)
0619 14:35:04.165715 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:04.165761 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 14:35:04.167014 (+ 1253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:35:04.167487 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:35:04.167493 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:35:04.167860 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:35:04.168349 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:35:04.168364 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:35:04.169117 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:35:04.169119 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:35:04.170037 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:35:04.170047 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:35:04.170510 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:35:04.170561 (+ 51us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:35:04.170706 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:35:04.171293 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:35:04.171309 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:35:04.172915 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:35:04.175001 (+ 2086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:35:04.175006 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:35:04.175021 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:35:04.175415 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:35:04.175676 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:35:04.175678 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:35:04.175679 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:35:04.175823 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:35:04.176307 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:35:04.176313 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:35:04.176564 (+ 251us) client_negotiation.cc:769] Sending connection context
0619 14:35:04.176750 (+ 186us) client_negotiation.cc:240] Negotiation successful
0619 14:35:04.177026 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":118,"threads_started":1}
I20260619 14:35:04.179222 17919 heartbeater.cc:347] Connected to a master server at 127.2.2.62:41891
I20260619 14:35:04.179476 17919 heartbeater.cc:464] Registering TS with master...
I20260619 14:35:04.180020 17919 heartbeater.cc:511] Master 127.2.2.62:41891 requested a full tablet report, sending...
I20260619 14:35:04.181061 17460 ts_manager.cc:194] Registered new tserver with Master: 8d1b5a5b42be462895944884b9163686 (127.2.2.3:45385)
I20260619 14:35:04.181680 17460 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.2.3@KRBTEST.COM'} at 127.2.2.3:49545
I20260619 14:35:04.187163 2056 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:35:04.196591 17499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:04.189123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53450 (local address 127.2.2.62:41891)
0619 14:35:04.189270 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 14:35:04.189275 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:35:04.189463 (+ 188us) server_negotiation.cc:407] Connection header received
0619 14:35:04.189604 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:35:04.189607 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:35:04.189651 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:35:04.189730 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:35:04.190488 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:35:04.190996 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:35:04.191804 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:35:04.192065 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:35:04.192163 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:35:04.193524 (+ 1361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:35:04.193527 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:35:04.193529 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:35:04.193550 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:35:04.195090 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:35:04.195505 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:35:04.195508 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:35:04.195509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:35:04.195578 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:35:04.195823 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:35:04.195825 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:35:04.195826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:35:04.196037 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:35:04.196139 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 14:35:04.196307 (+ 168us) server_negotiation.cc:299] Negotiation successful
0619 14:35:04.196396 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 14:35:04.199551 17460 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53450:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:35:04.202785 17460 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:35:04.217769 17933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:04.211526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.1:36581 (local address 127.0.0.1:32908)
0619 14:35:04.212471 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:04.212522 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 14:35:04.212654 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:35:04.213262 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:35:04.213266 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:35:04.213287 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:35:04.213397 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:35:04.213409 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:35:04.214669 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:35:04.214672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:35:04.216103 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:35:04.216113 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:35:04.217213 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:35:04.217216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:35:04.217337 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:35:04.217374 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 14:35:04.217491 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 14:35:04.217619 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":766,"mutex_wait_us":28,"thread_start_us":96,"threads_started":1}
I20260619 14:35:04.217777 17932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:04.211671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32908 (local address 127.2.2.1:36581)
0619 14:35:04.212109 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0619 14:35:04.212137 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0619 14:35:04.212802 (+ 665us) server_negotiation.cc:407] Connection header received
0619 14:35:04.212886 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:35:04.212892 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:35:04.213111 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:35:04.213277 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:35:04.213533 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:35:04.214559 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:35:04.216234 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:35:04.217098 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:35:04.217213 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:35:04.217372 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 14:35:04.217522 (+ 150us) server_negotiation.cc:299] Negotiation successful
0619 14:35:04.217619 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":114,"threads_started":1}
I20260619 14:35:04.218698 17931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:04.211586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.3:45385 (local address 127.0.0.1:44474)
0619 14:35:04.213191 (+ 1605us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:04.213207 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:35:04.213281 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:35:04.214085 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:35:04.214088 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:35:04.214106 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:35:04.214232 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:35:04.214238 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:35:04.215670 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:35:04.215673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:35:04.216658 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:35:04.216666 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:35:04.218261 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:35:04.218265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:35:04.218362 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:35:04.218387 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 14:35:04.218469 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 14:35:04.218562 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1533,"mutex_wait_us":69,"thread_start_us":101,"threads_started":1}
I20260619 14:35:04.218762 17934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:04.212330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44474 (local address 127.2.2.3:45385)
0619 14:35:04.213676 (+ 1346us) server_negotiation.cc:206] Beginning negotiation
0619 14:35:04.213691 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:35:04.213714 (+ 23us) server_negotiation.cc:407] Connection header received
0619 14:35:04.213779 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:35:04.213784 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:35:04.213943 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:35:04.214084 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:35:04.214365 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:35:04.215519 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:35:04.216771 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:35:04.217553 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:35:04.217664 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:35:04.217798 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 14:35:04.218516 (+ 718us) server_negotiation.cc:299] Negotiation successful
0619 14:35:04.218618 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1202,"thread_start_us":132,"threads_started":1}
I20260619 14:35:04.218802 17935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:04.212644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40960 (local address 127.2.2.2:33101)
0619 14:35:04.213047 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0619 14:35:04.213058 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:35:04.213534 (+ 476us) server_negotiation.cc:407] Connection header received
0619 14:35:04.213749 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:35:04.213754 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:35:04.213943 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:35:04.214068 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:35:04.214545 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:35:04.215509 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:35:04.216981 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:35:04.217776 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:35:04.218097 (+ 321us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:35:04.218258 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 14:35:04.218550 (+ 292us) server_negotiation.cc:299] Negotiation successful
0619 14:35:04.218644 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":93,"threads_started":1}
I20260619 14:35:04.219175 17930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:04.212088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.2:33101 (local address 127.0.0.1:40960)
0619 14:35:04.213444 (+ 1356us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:04.213458 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:35:04.213601 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:35:04.214316 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:35:04.214320 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:35:04.214339 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:35:04.214430 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:35:04.214436 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:35:04.215639 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:35:04.215643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:35:04.216843 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:35:04.216853 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:35:04.218308 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:35:04.218311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:35:04.218392 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:35:04.218413 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 14:35:04.218477 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 14:35:04.219052 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1297,"mutex_wait_us":497,"thread_start_us":108,"threads_started":1}
I20260619 14:35:04.220979 17585 tablet_service.cc:1467] Processing CreateTablet for tablet b3f91004a80a4c11af252f986e889c77 (DEFAULT_TABLE table=test-table [id=e337079b48b945cbb983a39b2a333588]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:35:04.221943 17585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3f91004a80a4c11af252f986e889c77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:35:04.221899 17719 tablet_service.cc:1467] Processing CreateTablet for tablet b3f91004a80a4c11af252f986e889c77 (DEFAULT_TABLE table=test-table [id=e337079b48b945cbb983a39b2a333588]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:35:04.221901 17853 tablet_service.cc:1467] Processing CreateTablet for tablet b3f91004a80a4c11af252f986e889c77 (DEFAULT_TABLE table=test-table [id=e337079b48b945cbb983a39b2a333588]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:35:04.222950 17719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3f91004a80a4c11af252f986e889c77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:35:04.222955 17853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3f91004a80a4c11af252f986e889c77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:35:04.228808 17936 tablet_bootstrap.cc:492] T b3f91004a80a4c11af252f986e889c77 P ed48fbe415dd43babd6582e39e83e02c: Bootstrap starting.
I20260619 14:35:04.231029 17938 tablet_bootstrap.cc:492] T b3f91004a80a4c11af252f986e889c77 P 37ea989b097c44838b4491edae66ac67: Bootstrap starting.
I20260619 14:35:04.231339 17936 tablet_bootstrap.cc:654] T b3f91004a80a4c11af252f986e889c77 P ed48fbe415dd43babd6582e39e83e02c: Neither blocks nor log segments found. Creating new log.
I20260619 14:35:04.232056 17936 log.cc:824] T b3f91004a80a4c11af252f986e889c77 P ed48fbe415dd43babd6582e39e83e02c: Log is configured to *not* fsync() on all Append() calls
I20260619 14:35:04.232903 17938 tablet_bootstrap.cc:654] T b3f91004a80a4c11af252f986e889c77 P 37ea989b097c44838b4491edae66ac67: Neither blocks nor log segments found. Creating new log.
I20260619 14:35:04.233879 17936 tablet_bootstrap.cc:492] T b3f91004a80a4c11af252f986e889c77 P ed48fbe415dd43babd6582e39e83e02c: No bootstrap required, opened a new log
I20260619 14:35:04.233879 17938 log.cc:824] T b3f91004a80a4c11af252f986e889c77 P 37ea989b097c44838b4491edae66ac67: Log is configured to *not* fsync() on all Append() calls
I20260619 14:35:04.234083 17936 ts_tablet_manager.cc:1397] T b3f91004a80a4c11af252f986e889c77 P ed48fbe415dd43babd6582e39e83e02c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:35:04.234514 17937 tablet_bootstrap.cc:492] T b3f91004a80a4c11af252f986e889c77 P 8d1b5a5b42be462895944884b9163686: Bootstrap starting.
I20260619 14:35:04.236044 17938 tablet_bootstrap.cc:492] T b3f91004a80a4c11af252f986e889c77 P 37ea989b097c44838b4491edae66ac67: No bootstrap required, opened a new log
I20260619 14:35:04.236232 17938 ts_tablet_manager.cc:1397] T b3f91004a80a4c11af252f986e889c77 P 37ea989b097c44838b4491edae66ac67: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:35:04.236693 17937 tablet_bootstrap.cc:654] T b3f91004a80a4c11af252f986e889c77 P 8d1b5a5b42be462895944884b9163686: Neither blocks nor log segments found. Creating new log.
I20260619 14:35:04.237864 17937 log.cc:824] T b3f91004a80a4c11af252f986e889c77 P 8d1b5a5b42be462895944884b9163686: Log is configured to *not* fsync() on all Append() calls
I20260619 14:35:04.238684 17936 raft_consensus.cc:348] T b3f91004a80a4c11af252f986e889c77 P ed48fbe415dd43babd6582e39e83e02c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed48fbe415dd43babd6582e39e83e02c" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 36581 } } peers { permanent_uuid: "8d1b5a5b42be462895944884b9163686" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45385 } } peers { permanent_uuid: "37ea989b097c44838b4491edae66ac67" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 33101 } }
I20260619 14:35:04.238977 17936 raft_consensus.cc:374] T b3f91004a80a4c11af252f986e889c77 P ed48fbe415dd43babd6582e39e83e02c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:35:04.239039 17936 raft_consensus.cc:729] T b3f91004a80a4c11af252f986e889c77 P ed48fbe415dd43babd6582e39e83e02c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed48fbe415dd43babd6582e39e83e02c, State: Initialized, Role: FOLLOWER
I20260619 14:35:04.239490 17936 consensus_queue.cc:260] T b3f91004a80a4c11af252f986e889c77 P ed48fbe415dd43babd6582e39e83e02c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed48fbe415dd43babd6582e39e83e02c" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 36581 } } peers { permanent_uuid: "8d1b5a5b42be462895944884b9163686" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45385 } } peers { permanent_uuid: "37ea989b097c44838b4491edae66ac67" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 33101 } }
I20260619 14:35:04.239508 17938 raft_consensus.cc:348] T b3f91004a80a4c11af252f986e889c77 P 37ea989b097c44838b4491edae66ac67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed48fbe415dd43babd6582e39e83e02c" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 36581 } } peers { permanent_uuid: "8d1b5a5b42be462895944884b9163686" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45385 } } peers { permanent_uuid: "37ea989b097c44838b4491edae66ac67" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 33101 } }
I20260619 14:35:04.239746 17938 raft_consensus.cc:374] T b3f91004a80a4c11af252f986e889c77 P 37ea989b097c44838b4491edae66ac67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:35:04.239809 17938 raft_consensus.cc:729] T b3f91004a80a4c11af252f986e889c77 P 37ea989b097c44838b4491edae66ac67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37ea989b097c44838b4491edae66ac67, State: Initialized, Role: FOLLOWER
I20260619 14:35:04.239940 17937 tablet_bootstrap.cc:492] T b3f91004a80a4c11af252f986e889c77 P 8d1b5a5b42be462895944884b9163686: No bootstrap required, opened a new log
I20260619 14:35:04.240123 17937 ts_tablet_manager.cc:1397] T b3f91004a80a4c11af252f986e889c77 P 8d1b5a5b42be462895944884b9163686: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260619 14:35:04.240304 17651 heartbeater.cc:503] Master 127.2.2.62:41891 was elected leader, sending a full tablet report...
I20260619 14:35:04.240515 17936 ts_tablet_manager.cc:1428] T b3f91004a80a4c11af252f986e889c77 P ed48fbe415dd43babd6582e39e83e02c: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260619 14:35:04.241448 17938 consensus_queue.cc:260] T b3f91004a80a4c11af252f986e889c77 P 37ea989b097c44838b4491edae66ac67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed48fbe415dd43babd6582e39e83e02c" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 36581 } } peers { permanent_uuid: "8d1b5a5b42be462895944884b9163686" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45385 } } peers { permanent_uuid: "37ea989b097c44838b4491edae66ac67" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 33101 } }
I20260619 14:35:04.242163 17785 heartbeater.cc:503] Master 127.2.2.62:41891 was elected leader, sending a full tablet report...
I20260619 14:35:04.243477 17938 ts_tablet_manager.cc:1428] T b3f91004a80a4c11af252f986e889c77 P 37ea989b097c44838b4491edae66ac67: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260619 14:35:04.244766 17937 raft_consensus.cc:348] T b3f91004a80a4c11af252f986e889c77 P 8d1b5a5b42be462895944884b9163686 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed48fbe415dd43babd6582e39e83e02c" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 36581 } } peers { permanent_uuid: "8d1b5a5b42be462895944884b9163686" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45385 } } peers { permanent_uuid: "37ea989b097c44838b4491edae66ac67" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 33101 } }
I20260619 14:35:04.245072 17937 raft_consensus.cc:374] T b3f91004a80a4c11af252f986e889c77 P 8d1b5a5b42be462895944884b9163686 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:35:04.245132 17937 raft_consensus.cc:729] T b3f91004a80a4c11af252f986e889c77 P 8d1b5a5b42be462895944884b9163686 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d1b5a5b42be462895944884b9163686, State: Initialized, Role: FOLLOWER
I20260619 14:35:04.245609 17937 consensus_queue.cc:260] T b3f91004a80a4c11af252f986e889c77 P 8d1b5a5b42be462895944884b9163686 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed48fbe415dd43babd6582e39e83e02c" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 36581 } } peers { permanent_uuid: "8d1b5a5b42be462895944884b9163686" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45385 } } peers { permanent_uuid: "37ea989b097c44838b4491edae66ac67" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 33101 } }
I20260619 14:35:04.246260 17919 heartbeater.cc:503] Master 127.2.2.62:41891 was elected leader, sending a full tablet report...
I20260619 14:35:04.246495 17937 ts_tablet_manager.cc:1428] T b3f91004a80a4c11af252f986e889c77 P 8d1b5a5b42be462895944884b9163686: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260619 14:35:04.353161 17942 raft_consensus.cc:482] T b3f91004a80a4c11af252f986e889c77 P ed48fbe415dd43babd6582e39e83e02c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:35:04.353430 17942 raft_consensus.cc:504] T b3f91004a80a4c11af252f986e889c77 P ed48fbe415dd43babd6582e39e83e02c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed48fbe415dd43babd6582e39e83e02c" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 36581 } } peers { permanent_uuid: "8d1b5a5b42be462895944884b9163686" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45385 } } peers { permanent_uuid: "37ea989b097c44838b4491edae66ac67" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 33101 } }
I20260619 14:35:04.354759 17942 leader_election.cc:288] T b3f91004a80a4c11af252f986e889c77 P ed48fbe415dd43babd6582e39e83e02c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d1b5a5b42be462895944884b9163686 (127.2.2.3:45385), 37ea989b097c44838b4491edae66ac67 (127.2.2.2:33101)
I20260619 14:35:04.360524 17934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:04.355200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:47061 (local address 127.2.2.3:45385)
0619 14:35:04.355332 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 14:35:04.355337 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:35:04.355662 (+ 325us) server_negotiation.cc:407] Connection header received
0619 14:35:04.355800 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:35:04.355803 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:35:04.355861 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:35:04.355952 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:35:04.356317 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:35:04.357101 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:35:04.359114 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:35:04.359779 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:35:04.359886 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:35:04.359955 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 14:35:04.360303 (+ 348us) server_negotiation.cc:299] Negotiation successful
0619 14:35:04.360385 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 14:35:04.360524 17935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:04.355201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:37421 (local address 127.2.2.2:33101)
0619 14:35:04.355332 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 14:35:04.355337 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:35:04.355733 (+ 396us) server_negotiation.cc:407] Connection header received
0619 14:35:04.355799 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:35:04.355802 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:35:04.355861 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:35:04.355953 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:35:04.356316 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:35:04.357098 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:35:04.359114 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:35:04.359777 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:35:04.359886 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:35:04.359955 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 14:35:04.360319 (+ 364us) server_negotiation.cc:299] Negotiation successful
0619 14:35:04.360386 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 14:35:04.360679 17946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:04.355096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.2:33101 (local address 127.2.2.1:37421)
0619 14:35:04.355616 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:04.355629 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:35:04.355711 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:35:04.355975 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:35:04.355978 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:35:04.355995 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:35:04.356198 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:35:04.356203 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:35:04.357207 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:35:04.357210 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:35:04.358970 (+ 1760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:35:04.358983 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:35:04.359915 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:35:04.359919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:35:04.360137 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:35:04.360190 (+ 53us) client_negotiation.cc:769] Sending connection context
0619 14:35:04.360264 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 14:35:04.360418 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"mutex_wait_us":379,"thread_start_us":100,"threads_started":1}
I20260619 14:35:04.361022 17945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:04.355096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.3:45385 (local address 127.2.2.1:47061)
0619 14:35:04.355523 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:04.355539 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:35:04.355687 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:35:04.355985 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:35:04.355987 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:35:04.356007 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:35:04.356185 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:35:04.356194 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:35:04.357207 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:35:04.357210 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:35:04.358970 (+ 1760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:35:04.358983 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:35:04.359915 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:35:04.359918 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:35:04.360142 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:35:04.360184 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 14:35:04.360264 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 14:35:04.360417 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":316,"spinlock_wait_cycles":1280,"thread_start_us":127,"threads_started":1}
I20260619 14:35:04.361439 17739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3f91004a80a4c11af252f986e889c77" candidate_uuid: "ed48fbe415dd43babd6582e39e83e02c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37ea989b097c44838b4491edae66ac67" is_pre_election: true
I20260619 14:35:04.361729 17873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3f91004a80a4c11af252f986e889c77" candidate_uuid: "ed48fbe415dd43babd6582e39e83e02c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d1b5a5b42be462895944884b9163686" is_pre_election: true
I20260619 14:35:04.361807 17739 raft_consensus.cc:2454] T b3f91004a80a4c11af252f986e889c77 P 37ea989b097c44838b4491edae66ac67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed48fbe415dd43babd6582e39e83e02c in term 0.
I20260619 14:35:04.361927 17944 raft_consensus.cc:482] T b3f91004a80a4c11af252f986e889c77 P 8d1b5a5b42be462895944884b9163686 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:35:04.362090 17944 raft_consensus.cc:504] T b3f91004a80a4c11af252f986e889c77 P 8d1b5a5b42be462895944884b9163686 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed48fbe415dd43babd6582e39e83e02c" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 36581 } } peers { permanent_uuid: "8d1b5a5b42be462895944884b9163686" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45385 } } peers { permanent_uuid: "37ea989b097c44838b4491edae66ac67" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 33101 } }
I20260619 14:35:04.362332 17539 leader_election.cc:302] T b3f91004a80a4c11af252f986e889c77 P ed48fbe415dd43babd6582e39e83e02c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37ea989b097c44838b4491edae66ac67, ed48fbe415dd43babd6582e39e83e02c; no voters:
I20260619 14:35:04.362581 17873 raft_consensus.cc:2454] T b3f91004a80a4c11af252f986e889c77 P 8d1b5a5b42be462895944884b9163686 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed48fbe415dd43babd6582e39e83e02c in term 0.
I20260619 14:35:04.362635 17942 raft_consensus.cc:2781] T b3f91004a80a4c11af252f986e889c77 P ed48fbe415dd43babd6582e39e83e02c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:35:04.362735 17942 raft_consensus.cc:482] T b3f91004a80a4c11af252f986e889c77 P ed48fbe415dd43babd6582e39e83e02c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:35:04.362789 17942 raft_consensus.cc:3037] T b3f91004a80a4c11af252f986e889c77 P ed48fbe415dd43babd6582e39e83e02c [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:35:04.363420 17944 leader_election.cc:288] T b3f91004a80a4c11af252f986e889c77 P 8d1b5a5b42be462895944884b9163686 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed48fbe415dd43babd6582e39e83e02c (127.2.2.1:36581), 37ea989b097c44838b4491edae66ac67 (127.2.2.2:33101)
I20260619 14:35:04.364274 17942 raft_consensus.cc:504] T b3f91004a80a4c11af252f986e889c77 P ed48fbe415dd43babd6582e39e83e02c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed48fbe415dd43babd6582e39e83e02c" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 36581 } } peers { permanent_uuid: "8d1b5a5b42be462895944884b9163686" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45385 } } peers { permanent_uuid: "37ea989b097c44838b4491edae66ac67" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 33101 } }
I20260619 14:35:04.364652 17942 leader_election.cc:288] T b3f91004a80a4c11af252f986e889c77 P ed48fbe415dd43babd6582e39e83e02c [CANDIDATE]: Term 1 election: Requested vote from peers 8d1b5a5b42be462895944884b9163686 (127.2.2.3:45385), 37ea989b097c44838b4491edae66ac67 (127.2.2.2:33101)
I20260619 14:35:04.365324 17873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3f91004a80a4c11af252f986e889c77" candidate_uuid: "ed48fbe415dd43babd6582e39e83e02c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d1b5a5b42be462895944884b9163686"
I20260619 14:35:04.365343 17739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3f91004a80a4c11af252f986e889c77" candidate_uuid: "ed48fbe415dd43babd6582e39e83e02c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37ea989b097c44838b4491edae66ac67"
I20260619 14:35:04.365447 17873 raft_consensus.cc:3037] T b3f91004a80a4c11af252f986e889c77 P 8d1b5a5b42be462895944884b9163686 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:35:04.365473 17739 raft_consensus.cc:3037] T b3f91004a80a4c11af252f986e889c77 P 37ea989b097c44838b4491edae66ac67 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:35:04.366819 17739 raft_consensus.cc:2454] T b3f91004a80a4c11af252f986e889c77 P 37ea989b097c44838b4491edae66ac67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed48fbe415dd43babd6582e39e83e02c in term 1.
I20260619 14:35:04.366828 17873 raft_consensus.cc:2454] T b3f91004a80a4c11af252f986e889c77 P 8d1b5a5b42be462895944884b9163686 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed48fbe415dd43babd6582e39e83e02c in term 1.
I20260619 14:35:04.367306 17541 leader_election.cc:302] T b3f91004a80a4c11af252f986e889c77 P ed48fbe415dd43babd6582e39e83e02c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d1b5a5b42be462895944884b9163686, ed48fbe415dd43babd6582e39e83e02c; no voters:
I20260619 14:35:04.367548 17942 raft_consensus.cc:2781] T b3f91004a80a4c11af252f986e889c77 P ed48fbe415dd43babd6582e39e83e02c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:35:04.367805 17942 raft_consensus.cc:686] T b3f91004a80a4c11af252f986e889c77 P ed48fbe415dd43babd6582e39e83e02c [term 1 LEADER]: Becoming Leader. State: Replica: ed48fbe415dd43babd6582e39e83e02c, State: Running, Role: LEADER
I20260619 14:35:04.368060 17932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:04.364007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.3:53401 (local address 127.2.2.1:36581)
0619 14:35:04.364123 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 14:35:04.364129 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:35:04.364143 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:35:04.364297 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:35:04.364301 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:35:04.364352 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:35:04.364444 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:35:04.364810 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:35:04.365491 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:35:04.367061 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:35:04.367536 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:35:04.367647 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:35:04.367710 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 14:35:04.367876 (+ 166us) server_negotiation.cc:299] Negotiation successful
0619 14:35:04.367946 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 14:35:04.368084 17947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:04.363592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.1:36581 (local address 127.2.2.3:53401)
0619 14:35:04.364082 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:04.364094 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:35:04.364181 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:35:04.364556 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:35:04.364559 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:35:04.364576 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:35:04.364710 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:35:04.364718 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:35:04.365626 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:35:04.365630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:35:04.366948 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:35:04.366960 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:35:04.367647 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:35:04.367651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:35:04.367748 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:35:04.367769 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 14:35:04.367862 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 14:35:04.367940 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"mutex_wait_us":83,"thread_start_us":122,"threads_started":1}
I20260619 14:35:04.368189 17942 consensus_queue.cc:237] T b3f91004a80a4c11af252f986e889c77 P ed48fbe415dd43babd6582e39e83e02c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed48fbe415dd43babd6582e39e83e02c" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 36581 } } peers { permanent_uuid: "8d1b5a5b42be462895944884b9163686" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45385 } } peers { permanent_uuid: "37ea989b097c44838b4491edae66ac67" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 33101 } }
I20260619 14:35:04.368752 17605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3f91004a80a4c11af252f986e889c77" candidate_uuid: "8d1b5a5b42be462895944884b9163686" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed48fbe415dd43babd6582e39e83e02c" is_pre_election: true
I20260619 14:35:04.371289 17921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:04.363592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.2:33101 (local address 127.2.2.3:45391)
0619 14:35:04.363708 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:04.363726 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:35:04.363819 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:35:04.364567 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:35:04.364569 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:35:04.364585 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:35:04.364721 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:35:04.364728 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:35:04.365936 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:35:04.365940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:35:04.367088 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:35:04.367101 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:35:04.370883 (+ 3782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:35:04.370887 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:35:04.371005 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:35:04.371029 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 14:35:04.371109 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 14:35:04.371195 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":3}
I20260619 14:35:04.371618 17935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:04.363853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.3:45391 (local address 127.2.2.2:33101)
0619 14:35:04.364280 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0619 14:35:04.364284 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:35:04.364296 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:35:04.364340 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:35:04.364342 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:35:04.364397 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:35:04.364467 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:35:04.365053 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:35:04.365789 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:35:04.370079 (+ 4290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:35:04.370748 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:35:04.370865 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:35:04.370927 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 14:35:04.371466 (+ 539us) server_negotiation.cc:299] Negotiation successful
0619 14:35:04.371537 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308}
I20260619 14:35:04.371984 17739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3f91004a80a4c11af252f986e889c77" candidate_uuid: "8d1b5a5b42be462895944884b9163686" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37ea989b097c44838b4491edae66ac67" is_pre_election: true
I20260619 14:35:04.372164 17739 raft_consensus.cc:2379] T b3f91004a80a4c11af252f986e889c77 P 37ea989b097c44838b4491edae66ac67 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8d1b5a5b42be462895944884b9163686 in current term 1: Already voted for candidate ed48fbe415dd43babd6582e39e83e02c in this term.
I20260619 14:35:04.371827 17460 catalog_manager.cc:5495] T b3f91004a80a4c11af252f986e889c77 P ed48fbe415dd43babd6582e39e83e02c reported cstate change: term changed from 0 to 1, leader changed from <none> to ed48fbe415dd43babd6582e39e83e02c (127.2.2.1). New cstate: current_term: 1 leader_uuid: "ed48fbe415dd43babd6582e39e83e02c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed48fbe415dd43babd6582e39e83e02c" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 36581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d1b5a5b42be462895944884b9163686" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37ea989b097c44838b4491edae66ac67" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 33101 } health_report { overall_health: UNKNOWN } } }
I20260619 14:35:04.372628 17807 leader_election.cc:302] T b3f91004a80a4c11af252f986e889c77 P 8d1b5a5b42be462895944884b9163686 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d1b5a5b42be462895944884b9163686; no voters: 37ea989b097c44838b4491edae66ac67, ed48fbe415dd43babd6582e39e83e02c
I20260619 14:35:04.372853 17944 raft_consensus.cc:2726] T b3f91004a80a4c11af252f986e889c77 P 8d1b5a5b42be462895944884b9163686 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260619 14:35:04.414047 17920 tablet.cc:2369] T b3f91004a80a4c11af252f986e889c77 P 8d1b5a5b42be462895944884b9163686: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:35:04.415743 17932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:04.411919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32916 (local address 127.2.2.1:36581)
0619 14:35:04.412047 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 14:35:04.412051 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:35:04.412064 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:35:04.412200 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:35:04.412202 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:35:04.412260 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:35:04.412347 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:35:04.412628 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:35:04.413150 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:35:04.414264 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:35:04.414572 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:35:04.414691 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:35:04.414957 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:35:04.415358 (+ 401us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:35:04.415450 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 14:35:04.415521 (+ 71us) server_negotiation.cc:299] Negotiation successful
0619 14:35:04.415572 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 14:35:04.421587 17739 raft_consensus.cc:1261] T b3f91004a80a4c11af252f986e889c77 P 37ea989b097c44838b4491edae66ac67 [term 1 FOLLOWER]: Refusing update from remote peer ed48fbe415dd43babd6582e39e83e02c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:35:04.421624 17873 raft_consensus.cc:1261] T b3f91004a80a4c11af252f986e889c77 P 8d1b5a5b42be462895944884b9163686 [term 1 FOLLOWER]: Refusing update from remote peer ed48fbe415dd43babd6582e39e83e02c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:35:04.422295 17948 consensus_queue.cc:1035] T b3f91004a80a4c11af252f986e889c77 P ed48fbe415dd43babd6582e39e83e02c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d1b5a5b42be462895944884b9163686" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:35:04.422492 17942 consensus_queue.cc:1035] T b3f91004a80a4c11af252f986e889c77 P ed48fbe415dd43babd6582e39e83e02c [LEADER]: Connected to new peer: Peer: permanent_uuid: "37ea989b097c44838b4491edae66ac67" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 33101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:35:04.430462 17952 mvcc.cc:205] Tried to move back new op lower bound from 7298579269303193600 to 7298579269095477248. Current Snapshot: MvccSnapshot[applied={T|T < 7298579269303193600}]
I20260619 14:35:04.438871 17459 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:35:04.439088 17459 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:35:04.441491 17459 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f: Sending DeleteTablet for 3 replicas of tablet b3f91004a80a4c11af252f986e889c77
I20260619 14:35:04.442276 17585 tablet_service.cc:1514] Processing DeleteTablet for tablet b3f91004a80a4c11af252f986e889c77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:35:04 UTC) from {username='kudu', principal='kudu/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:32908
I20260619 14:35:04.442288 17853 tablet_service.cc:1514] Processing DeleteTablet for tablet b3f91004a80a4c11af252f986e889c77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:35:04 UTC) from {username='kudu', principal='kudu/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:44474
I20260619 14:35:04.442842 17961 tablet_replica.cc:330] T b3f91004a80a4c11af252f986e889c77 P ed48fbe415dd43babd6582e39e83e02c: stopping tablet replica
I20260619 14:35:04.443019 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 17522
I20260619 14:35:04.443132 17719 tablet_service.cc:1514] Processing DeleteTablet for tablet b3f91004a80a4c11af252f986e889c77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:35:04 UTC) from {username='kudu', principal='kudu/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:40960
I20260619 14:35:04.444006 17962 tablet_replica.cc:330] T b3f91004a80a4c11af252f986e889c77 P 8d1b5a5b42be462895944884b9163686: stopping tablet replica
I20260619 14:35:04.444278 17962 raft_consensus.cc:2229] T b3f91004a80a4c11af252f986e889c77 P 8d1b5a5b42be462895944884b9163686 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:35:04.444568 17962 raft_consensus.cc:2258] T b3f91004a80a4c11af252f986e889c77 P 8d1b5a5b42be462895944884b9163686 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:35:04.444929 17963 tablet_replica.cc:330] T b3f91004a80a4c11af252f986e889c77 P 37ea989b097c44838b4491edae66ac67: stopping tablet replica
I20260619 14:35:04.445196 17963 raft_consensus.cc:2229] T b3f91004a80a4c11af252f986e889c77 P 37ea989b097c44838b4491edae66ac67 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:35:04.445484 17963 raft_consensus.cc:2258] T b3f91004a80a4c11af252f986e889c77 P 37ea989b097c44838b4491edae66ac67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:35:04.446286 17962 ts_tablet_manager.cc:1905] T b3f91004a80a4c11af252f986e889c77 P 8d1b5a5b42be462895944884b9163686: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:35:04.446655 17963 ts_tablet_manager.cc:1905] T b3f91004a80a4c11af252f986e889c77 P 37ea989b097c44838b4491edae66ac67: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:35:04.448498 17962 ts_tablet_manager.cc:1918] T b3f91004a80a4c11af252f986e889c77 P 8d1b5a5b42be462895944884b9163686: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:35:04.448601 17962 log.cc:1196] T b3f91004a80a4c11af252f986e889c77 P 8d1b5a5b42be462895944884b9163686: Deleting WAL directory at /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-2/wal/wals/b3f91004a80a4c11af252f986e889c77
I20260619 14:35:04.449141 17962 ts_tablet_manager.cc:1939] T b3f91004a80a4c11af252f986e889c77 P 8d1b5a5b42be462895944884b9163686: Deleting consensus metadata
I20260619 14:35:04.449281 17963 ts_tablet_manager.cc:1918] T b3f91004a80a4c11af252f986e889c77 P 37ea989b097c44838b4491edae66ac67: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:35:04.449383 17963 log.cc:1196] T b3f91004a80a4c11af252f986e889c77 P 37ea989b097c44838b4491edae66ac67: Deleting WAL directory at /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-1/wal/wals/b3f91004a80a4c11af252f986e889c77
I20260619 14:35:04.449690 17963 ts_tablet_manager.cc:1939] T b3f91004a80a4c11af252f986e889c77 P 37ea989b097c44838b4491edae66ac67: Deleting consensus metadata
W20260619 14:35:04.450317 17444 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.2.1:36581 (error 108)
I20260619 14:35:04.450453 17447 catalog_manager.cc:4841] TS 8d1b5a5b42be462895944884b9163686 (127.2.2.3:45385): tablet b3f91004a80a4c11af252f986e889c77 (table test-table [id=e337079b48b945cbb983a39b2a333588]) successfully deleted
I20260619 14:35:04.450534 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 17656
I20260619 14:35:04.451894 17930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:04.451635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.1:36581 (local address 127.0.0.1:32920)
0619 14:35:04.451756 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:04.451823 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.1:36581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 14:35:04.452086 17444 catalog_manager.cc:4584] TS ed48fbe415dd43babd6582e39e83e02c (127.2.2.1:36581): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b3f91004a80a4c11af252f986e889c77: Network error: Client connection negotiation failed: client connection to 127.2.2.1:36581: connect: Connection refused (error 111)
I20260619 14:35:04.456575 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 17790
I20260619 14:35:04.457396 17930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:04.456676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.2:33101 (local address 127.0.0.1:40976)
0619 14:35:04.457232 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:04.457296 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.2:33101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460}
I20260619 14:35:04.461985 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 17428
I20260619 14:35:24.468850 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:41891
--webserver_interface=127.2.2.62
--webserver_port=34103
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:37921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.2.62:41891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:35:24.578881 17965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:35:24.579164 17965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:35:24.579253 17965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:35:24.583549 17965 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 14:35:24.583611 17965 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 14:35:24.583626 17965 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 14:35:24.583642 17965 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:35:24.583660 17965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:35:24.583725 17965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:35:24.583751 17965 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:35:24.583782 17965 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:35:24.589671 17965 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:37921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.2.62:41891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:41891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.2.62
--webserver_port=34103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:35:24.590858 17965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:35:24.591956 17965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:35:24.605463 17965 server_base.cc:1034] running on GCE node
W20260619 14:35:24.605392 17971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:35:24.605407 17972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:35:24.605365 17974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:35:24.606273 17965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:35:24.607425 17965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:35:24.608611 17965 hybrid_clock.cc:648] HybridClock initialized: now 1781879724608595 us; error 31 us; skew 500 ppm
Jun 19 14:35:24 dist-test-slave-70rm krb5kdc[17415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879724, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:35:24.611531 17965 init.cc:377] Logged in from keytab as kudu/127.2.2.62@KRBTEST.COM (short username kudu)
I20260619 14:35:24.612463 17965 webserver.cc:460] Webserver started at http://127.2.2.62:34103/ using document root <none> and password file <none>
I20260619 14:35:24.613054 17965 fs_manager.cc:359] Metadata directory not provided
I20260619 14:35:24.613128 17965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:35:24.617238 17965 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:35:24.619663 17981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:35:24.621008 17965 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:35:24.621170 17965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/master-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/master-0/wal
uuid: "b57bdd13c44f4b00a3d67fc5b168cb3f"
format_stamp: "Formatted at 2026-06-19 14:35:03 on dist-test-slave-70rm"
server_key: "536d4638e100092d76e2479fb40d531e"
server_key_iv: "b9de7b5fc7fe0ee9164acb5b1e8cea70"
server_key_version: "encryptionkey@0"
I20260619 14:35:24.621815 17965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:35:24.638500 17965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:35:24.641602 17965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:35:24.641839 17965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:35:24.649233 17965 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.62:41891
I20260619 14:35:24.649536 18033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.62:41891 every 8 connection(s)
I20260619 14:35:24.650398 17965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
I20260619 14:35:24.654831 18034 sys_catalog.cc:263] Verifying existing consensus state
I20260619 14:35:24.655593 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 17965
I20260619 14:35:24.656015 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.1:36581
--local_ip_for_outbound_sockets=127.2.2.1
--tserver_master_addrs=127.2.2.62:41891
--webserver_port=43115
--webserver_interface=127.2.2.1
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab
--principal=kudu/127.2.2.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:37921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:35:24.656710 18034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f: Bootstrap starting.
I20260619 14:35:24.673969 18034 log.cc:824] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f: Log is configured to *not* fsync() on all Append() calls
I20260619 14:35:24.682928 18034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:35:24.683319 18034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f: Bootstrap complete.
I20260619 14:35:24.687489 18034 raft_consensus.cc:348] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57bdd13c44f4b00a3d67fc5b168cb3f" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41891 } }
I20260619 14:35:24.687983 18034 raft_consensus.cc:729] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b57bdd13c44f4b00a3d67fc5b168cb3f, State: Initialized, Role: FOLLOWER
I20260619 14:35:24.688468 18034 consensus_queue.cc:260] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57bdd13c44f4b00a3d67fc5b168cb3f" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41891 } }
I20260619 14:35:24.688611 18034 raft_consensus.cc:388] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:35:24.688706 18034 raft_consensus.cc:482] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:35:24.688819 18034 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f [term 1 FOLLOWER]: Advancing to term 2
I20260619 14:35:24.690893 18034 raft_consensus.cc:504] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57bdd13c44f4b00a3d67fc5b168cb3f" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41891 } }
I20260619 14:35:24.691248 18034 leader_election.cc:302] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b57bdd13c44f4b00a3d67fc5b168cb3f; no voters:
I20260619 14:35:24.691602 18034 leader_election.cc:288] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 14:35:24.691691 18038 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f [term 2 FOLLOWER]: Leader election won for term 2
I20260619 14:35:24.691895 18038 raft_consensus.cc:686] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f [term 2 LEADER]: Becoming Leader. State: Replica: b57bdd13c44f4b00a3d67fc5b168cb3f, State: Running, Role: LEADER
I20260619 14:35:24.692265 18038 consensus_queue.cc:237] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57bdd13c44f4b00a3d67fc5b168cb3f" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41891 } }
I20260619 14:35:24.692914 18034 sys_catalog.cc:564] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:35:24.693235 18038 sys_catalog.cc:455] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b57bdd13c44f4b00a3d67fc5b168cb3f. Latest consensus state: current_term: 2 leader_uuid: "b57bdd13c44f4b00a3d67fc5b168cb3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57bdd13c44f4b00a3d67fc5b168cb3f" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41891 } } }
I20260619 14:35:24.693449 18038 sys_catalog.cc:458] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f [sys.catalog]: This master's current role is: LEADER
I20260619 14:35:24.693616 18039 sys_catalog.cc:455] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b57bdd13c44f4b00a3d67fc5b168cb3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57bdd13c44f4b00a3d67fc5b168cb3f" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41891 } } }
I20260619 14:35:24.693799 18039 sys_catalog.cc:458] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f [sys.catalog]: This master's current role is: LEADER
I20260619 14:35:24.693892 18042 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:35:24.698820 18042 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:35:24.699512 18042 catalog_manager.cc:1259] Loaded cluster ID: f112e01fc9464ef8b0580b565920bf8a
I20260619 14:35:24.699576 18042 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:35:24.702060 18042 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:35:24.702823 18042 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f: Loaded TSK: 0
I20260619 14:35:24.732996 18042 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f: Generated new TSK 1
I20260619 14:35:24.733311 18042 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 14:35:24.770768 18036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:35:24.771067 18036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:35:24.771143 18036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:35:24.775234 18036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:35:24.775331 18036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:35:24.775422 18036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.1
I20260619 14:35:24.781167 18036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:37921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab
--principal=kudu/127.2.2.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.1:36581
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.2.1
--webserver_port=43115
--tserver_master_addrs=127.2.2.62:41891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.1
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:35:24.782632 18036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:35:24.783860 18036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:35:24.794298 18062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:35:24.794452 18064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:35:24.794312 18061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:35:24.794828 18036 server_base.cc:1034] running on GCE node
I20260619 14:35:24.795303 18036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:35:24.795974 18036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:35:24.797168 18036 hybrid_clock.cc:648] HybridClock initialized: now 1781879724797152 us; error 34 us; skew 500 ppm
Jun 19 14:35:24 dist-test-slave-70rm krb5kdc[17415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879724, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:35:24.799968 18036 init.cc:377] Logged in from keytab as kudu/127.2.2.1@KRBTEST.COM (short username kudu)
I20260619 14:35:24.800942 18036 webserver.cc:460] Webserver started at http://127.2.2.1:43115/ using document root <none> and password file <none>
I20260619 14:35:24.801514 18036 fs_manager.cc:359] Metadata directory not provided
I20260619 14:35:24.801587 18036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:35:24.805689 18036 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:35:24.808104 18071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:35:24.809489 18036 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:35:24.809631 18036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-0/wal
uuid: "ed48fbe415dd43babd6582e39e83e02c"
format_stamp: "Formatted at 2026-06-19 14:35:03 on dist-test-slave-70rm"
server_key: "063a84231a70ecfe86ea55c3b4623636"
server_key_iv: "f52d9d4620e8c4aacadf7d13af1f6ed4"
server_key_version: "encryptionkey@0"
I20260619 14:35:24.810261 18036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:35:24.831030 18036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:35:24.834543 18036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:35:24.834759 18036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:35:24.835389 18036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:35:24.836570 18078 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 14:35:24.840669 18036 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 14:35:24.840735 18036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.001s
I20260619 14:35:24.840812 18036 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 14:35:24.842505 18036 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 14:35:24.842562 18036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260619 14:35:24.842811 18078 tablet_bootstrap.cc:492] T b3f91004a80a4c11af252f986e889c77 P ed48fbe415dd43babd6582e39e83e02c: Bootstrap starting.
I20260619 14:35:24.853461 18036 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.1:36581
I20260619 14:35:24.853511 18185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.1:36581 every 8 connection(s)
I20260619 14:35:24.854723 18036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
I20260619 14:35:24.862962 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 18036
I20260619 14:35:24.863389 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.2:33101
--local_ip_for_outbound_sockets=127.2.2.2
--tserver_master_addrs=127.2.2.62:41891
--webserver_port=40855
--webserver_interface=127.2.2.2
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab
--principal=kudu/127.2.2.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:37921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:35:24.863550 18078 log.cc:824] T b3f91004a80a4c11af252f986e889c77 P ed48fbe415dd43babd6582e39e83e02c: Log is configured to *not* fsync() on all Append() calls
Jun 19 14:35:24 dist-test-slave-70rm krb5kdc[17415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879724, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.1@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:35:24.872535 18078 tablet_bootstrap.cc:492] T b3f91004a80a4c11af252f986e889c77 P ed48fbe415dd43babd6582e39e83e02c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:35:24.872828 18188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:24.856920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:45513 (local address 127.2.2.62:41891)
0619 14:35:24.857268 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0619 14:35:24.857289 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 14:35:24.857645 (+ 356us) server_negotiation.cc:407] Connection header received
0619 14:35:24.858818 (+ 1173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:35:24.858829 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:35:24.859120 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:35:24.859446 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:35:24.860562 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:35:24.861508 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:35:24.862797 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:35:24.863478 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:35:24.863654 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:35:24.866608 (+ 2954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:35:24.866614 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:35:24.866630 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:35:24.866664 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:35:24.868842 (+ 2178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:35:24.869594 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:35:24.869600 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:35:24.869606 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:35:24.869756 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:35:24.870289 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:35:24.870293 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:35:24.870294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:35:24.870731 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:35:24.870877 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 14:35:24.871610 (+ 733us) server_negotiation.cc:299] Negotiation successful
0619 14:35:24.871919 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":115,"threads_started":1}
I20260619 14:35:24.872923 18189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:24.856972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41891 (local address 127.2.2.1:45513)
0619 14:35:24.857476 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:24.857518 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 14:35:24.858403 (+ 885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:35:24.859499 (+ 1096us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:35:24.859507 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:35:24.859931 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:35:24.860389 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:35:24.860402 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:35:24.861672 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:35:24.861674 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:35:24.862635 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:35:24.862645 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:35:24.863752 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:35:24.863792 (+ 40us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:35:24.864034 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:35:24.864738 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:35:24.864767 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:35:24.866395 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:35:24.868991 (+ 2596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:35:24.869001 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:35:24.869018 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:35:24.869445 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:35:24.869886 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:35:24.869891 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:35:24.869893 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:35:24.870090 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:35:24.870881 (+ 791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:35:24.870891 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:35:24.871234 (+ 343us) client_negotiation.cc:769] Sending connection context
0619 14:35:24.871457 (+ 223us) client_negotiation.cc:240] Negotiation successful
0619 14:35:24.871874 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":136,"threads_started":1}
I20260619 14:35:24.872987 18078 tablet_bootstrap.cc:492] T b3f91004a80a4c11af252f986e889c77 P ed48fbe415dd43babd6582e39e83e02c: Bootstrap complete.
I20260619 14:35:24.874362 18078 ts_tablet_manager.cc:1397] T b3f91004a80a4c11af252f986e889c77 P ed48fbe415dd43babd6582e39e83e02c: Time spent bootstrapping tablet: real 0.032s user 0.027s sys 0.004s
I20260619 14:35:24.875741 18186 heartbeater.cc:347] Connected to a master server at 127.2.2.62:41891
I20260619 14:35:24.875983 18186 heartbeater.cc:464] Registering TS with master...
I20260619 14:35:24.876510 18186 heartbeater.cc:511] Master 127.2.2.62:41891 requested a full tablet report, sending...
I20260619 14:35:24.877010 18078 raft_consensus.cc:348] T b3f91004a80a4c11af252f986e889c77 P ed48fbe415dd43babd6582e39e83e02c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed48fbe415dd43babd6582e39e83e02c" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 36581 } } peers { permanent_uuid: "8d1b5a5b42be462895944884b9163686" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45385 } } peers { permanent_uuid: "37ea989b097c44838b4491edae66ac67" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 33101 } }
I20260619 14:35:24.877740 18078 raft_consensus.cc:729] T b3f91004a80a4c11af252f986e889c77 P ed48fbe415dd43babd6582e39e83e02c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed48fbe415dd43babd6582e39e83e02c, State: Initialized, Role: FOLLOWER
I20260619 14:35:24.878312 18078 consensus_queue.cc:260] T b3f91004a80a4c11af252f986e889c77 P ed48fbe415dd43babd6582e39e83e02c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed48fbe415dd43babd6582e39e83e02c" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 36581 } } peers { permanent_uuid: "8d1b5a5b42be462895944884b9163686" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45385 } } peers { permanent_uuid: "37ea989b097c44838b4491edae66ac67" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 33101 } }
I20260619 14:35:24.879328 18078 ts_tablet_manager.cc:1428] T b3f91004a80a4c11af252f986e889c77 P ed48fbe415dd43babd6582e39e83e02c: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:35:24.879462 17998 ts_manager.cc:194] Registered new tserver with Master: ed48fbe415dd43babd6582e39e83e02c (127.2.2.1:36581)
I20260619 14:35:24.883812 17998 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.2.1@KRBTEST.COM'} at 127.2.2.1:45513
I20260619 14:35:24.886812 18186 heartbeater.cc:503] Master 127.2.2.62:41891 was elected leader, sending a full tablet report...
Jun 19 14:35:24 dist-test-slave-70rm krb5kdc[17415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879724, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.62@KRBTEST.COM for kudu/127.2.2.1@KRBTEST.COM
I20260619 14:35:24.894903 18194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:24.882934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38340 (local address 127.2.2.1:36581)
0619 14:35:24.883839 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0619 14:35:24.883848 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:35:24.883865 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:35:24.883950 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:35:24.883956 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:35:24.884146 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:35:24.884302 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:35:24.885070 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:35:24.886081 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:35:24.887494 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:35:24.887982 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:35:24.888106 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:35:24.890736 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:35:24.890747 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:35:24.890761 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:35:24.890790 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:35:24.892742 (+ 1952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:35:24.893235 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:35:24.893263 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:35:24.893265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:35:24.893386 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:35:24.893719 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:35:24.893721 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:35:24.893722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:35:24.894045 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:35:24.894225 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0619 14:35:24.894587 (+ 362us) server_negotiation.cc:299] Negotiation successful
0619 14:35:24.894774 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":810,"thread_start_us":93,"threads_started":1}
I20260619 14:35:24.894965 18193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:24.882865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.1:36581 (local address 127.0.0.1:38340)
0619 14:35:24.883427 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:24.883474 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 14:35:24.883603 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:35:24.884303 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:35:24.884308 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:35:24.884797 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:35:24.884929 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:35:24.884937 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:35:24.886265 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:35:24.886268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:35:24.887362 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:35:24.887377 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:35:24.888164 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:35:24.888181 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:35:24.888305 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:35:24.888966 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:35:24.888981 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:35:24.890579 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:35:24.892859 (+ 2280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:35:24.892874 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:35:24.892880 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:35:24.893111 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:35:24.893469 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:35:24.893473 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:35:24.893475 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:35:24.893588 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:35:24.894193 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:35:24.894200 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:35:24.894343 (+ 143us) client_negotiation.cc:769] Sending connection context
0619 14:35:24.894522 (+ 179us) client_negotiation.cc:240] Negotiation successful
0619 14:35:24.894752 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":132,"threads_started":1}
I20260619 14:35:24.896013 18120 tablet_service.cc:1514] Processing DeleteTablet for tablet b3f91004a80a4c11af252f986e889c77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:35:04 UTC) from {username='kudu', principal='kudu/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:38340
I20260619 14:35:24.897073 18195 tablet_replica.cc:330] stopping tablet replica
I20260619 14:35:24.897405 18195 raft_consensus.cc:2229] T b3f91004a80a4c11af252f986e889c77 P ed48fbe415dd43babd6582e39e83e02c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:35:24.897670 18195 raft_consensus.cc:2258] T b3f91004a80a4c11af252f986e889c77 P ed48fbe415dd43babd6582e39e83e02c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:35:24.898733 18195 ts_tablet_manager.cc:1905] T b3f91004a80a4c11af252f986e889c77 P ed48fbe415dd43babd6582e39e83e02c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:35:24.902470 18195 ts_tablet_manager.cc:1918] T b3f91004a80a4c11af252f986e889c77 P ed48fbe415dd43babd6582e39e83e02c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:35:24.902601 18195 log.cc:1196] T b3f91004a80a4c11af252f986e889c77 P ed48fbe415dd43babd6582e39e83e02c: Deleting WAL directory at /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-0/wal/wals/b3f91004a80a4c11af252f986e889c77
I20260619 14:35:24.902884 18195 ts_tablet_manager.cc:1939] T b3f91004a80a4c11af252f986e889c77 P ed48fbe415dd43babd6582e39e83e02c: Deleting consensus metadata
I20260619 14:35:24.904975 17982 catalog_manager.cc:4841] TS ed48fbe415dd43babd6582e39e83e02c (127.2.2.1:36581): tablet b3f91004a80a4c11af252f986e889c77 (table test-table [id=e337079b48b945cbb983a39b2a333588]) successfully deleted
W20260619 14:35:24.985292 18191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:35:24.985559 18191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:35:24.985602 18191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:35:24.989905 18191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:35:24.989964 18191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:35:24.990024 18191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.2
I20260619 14:35:24.995527 18191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:37921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab
--principal=kudu/127.2.2.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.2:33101
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.2.2
--webserver_port=40855
--tserver_master_addrs=127.2.2.62:41891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.2
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:35:24.996454 18191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:35:24.997394 18191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:35:25.007462 18203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:35:25.007462 18200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:35:25.007565 18191 server_base.cc:1034] running on GCE node
W20260619 14:35:25.007474 18201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:35:25.008042 18191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:35:25.008755 18191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:35:25.009922 18191 hybrid_clock.cc:648] HybridClock initialized: now 1781879725009905 us; error 30 us; skew 500 ppm
Jun 19 14:35:25 dist-test-slave-70rm krb5kdc[17415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879725, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:35:25.012679 18191 init.cc:377] Logged in from keytab as kudu/127.2.2.2@KRBTEST.COM (short username kudu)
I20260619 14:35:25.013633 18191 webserver.cc:460] Webserver started at http://127.2.2.2:40855/ using document root <none> and password file <none>
I20260619 14:35:25.014258 18191 fs_manager.cc:359] Metadata directory not provided
I20260619 14:35:25.014356 18191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:35:25.018172 18191 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:35:25.020308 18210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:35:25.021505 18191 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:35:25.021631 18191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-1/wal
uuid: "37ea989b097c44838b4491edae66ac67"
format_stamp: "Formatted at 2026-06-19 14:35:03 on dist-test-slave-70rm"
server_key: "821e7ea015dc5edf0313fdc50aa93224"
server_key_iv: "80baa16d283bb6b366df3cf486b93e40"
server_key_version: "encryptionkey@0"
I20260619 14:35:25.022224 18191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:35:25.033818 18191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:35:25.037060 18191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:35:25.037264 18191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:35:25.037887 18191 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:35:25.038736 18191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:35:25.038782 18191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:35:25.038847 18191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:35:25.038889 18191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:35:25.047740 18191 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.2:33101
I20260619 14:35:25.047763 18323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.2:33101 every 8 connection(s)
I20260619 14:35:25.048791 18191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
I20260619 14:35:25.050954 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 18191
I20260619 14:35:25.051391 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.3:45385
--local_ip_for_outbound_sockets=127.2.2.3
--tserver_master_addrs=127.2.2.62:41891
--webserver_port=44157
--webserver_interface=127.2.2.3
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab
--principal=kudu/127.2.2.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:37921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:35:25 dist-test-slave-70rm krb5kdc[17415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879725, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.2@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:35:25.063848 18188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:25.050801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:35935 (local address 127.2.2.62:41891)
0619 14:35:25.050989 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 14:35:25.050994 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:35:25.052046 (+ 1052us) server_negotiation.cc:407] Connection header received
0619 14:35:25.053424 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:35:25.053428 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:35:25.053495 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:35:25.053603 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:35:25.054954 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:35:25.055504 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:35:25.056708 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:35:25.056943 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:35:25.057053 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:35:25.060172 (+ 3119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:35:25.060175 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:35:25.060177 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:35:25.060205 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:35:25.061729 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:35:25.062315 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:35:25.062319 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:35:25.062320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:35:25.062393 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:35:25.062738 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:35:25.062741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:35:25.062743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:35:25.062929 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:35:25.063030 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 14:35:25.063534 (+ 504us) server_negotiation.cc:299] Negotiation successful
0619 14:35:25.063670 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 14:35:25.064690 18326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:25.051263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41891 (local address 127.2.2.2:35935)
0619 14:35:25.051844 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:25.051900 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 14:35:25.053183 (+ 1283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:35:25.053805 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:35:25.053812 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:35:25.054266 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:35:25.054787 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:35:25.054802 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:35:25.055643 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:35:25.055646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:35:25.056555 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:35:25.056564 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:35:25.057041 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:35:25.057101 (+ 60us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:35:25.057280 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:35:25.058016 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:35:25.058036 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:35:25.059975 (+ 1939us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:35:25.061849 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:35:25.061855 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:35:25.061872 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:35:25.062189 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:35:25.062491 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:35:25.062494 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:35:25.062496 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:35:25.062638 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:35:25.063041 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:35:25.063048 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:35:25.063301 (+ 253us) client_negotiation.cc:769] Sending connection context
0619 14:35:25.063484 (+ 183us) client_negotiation.cc:240] Negotiation successful
0619 14:35:25.063832 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":148,"threads_started":1}
I20260619 14:35:25.065930 18324 heartbeater.cc:347] Connected to a master server at 127.2.2.62:41891
I20260619 14:35:25.066252 18324 heartbeater.cc:464] Registering TS with master...
I20260619 14:35:25.066748 18324 heartbeater.cc:511] Master 127.2.2.62:41891 requested a full tablet report, sending...
I20260619 14:35:25.067785 17998 ts_manager.cc:194] Registered new tserver with Master: 37ea989b097c44838b4491edae66ac67 (127.2.2.2:33101)
I20260619 14:35:25.068539 17998 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.2.2@KRBTEST.COM'} at 127.2.2.2:35935
W20260619 14:35:25.165791 18327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:35:25.166082 18327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:35:25.166150 18327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:35:25.170281 18327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:35:25.170352 18327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:35:25.170413 18327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.3
I20260619 14:35:25.175650 18327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:37921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab
--principal=kudu/127.2.2.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.3:45385
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.2.3
--webserver_port=44157
--tserver_master_addrs=127.2.2.62:41891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.3
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:35:25.176710 18327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:35:25.177809 18327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:35:25.187922 18332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:35:25.187942 18333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:35:25.187922 18335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:35:25.188238 18327 server_base.cc:1034] running on GCE node
I20260619 14:35:25.188908 18327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:35:25.189564 18327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:35:25.190809 18327 hybrid_clock.cc:648] HybridClock initialized: now 1781879725190786 us; error 26 us; skew 500 ppm
Jun 19 14:35:25 dist-test-slave-70rm krb5kdc[17415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879725, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:35:25.193748 18327 init.cc:377] Logged in from keytab as kudu/127.2.2.3@KRBTEST.COM (short username kudu)
I20260619 14:35:25.194877 18327 webserver.cc:460] Webserver started at http://127.2.2.3:44157/ using document root <none> and password file <none>
I20260619 14:35:25.195621 18327 fs_manager.cc:359] Metadata directory not provided
I20260619 14:35:25.195760 18327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:35:25.200035 18327 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:35:25.202553 18342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:35:25.203961 18327 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:35:25.204119 18327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-2/wal
uuid: "8d1b5a5b42be462895944884b9163686"
format_stamp: "Formatted at 2026-06-19 14:35:04 on dist-test-slave-70rm"
server_key: "01fabbf4a4bacd1958c69260620e5990"
server_key_iv: "255e76cd5e1a3ae84d6154042addcaa4"
server_key_version: "encryptionkey@0"
I20260619 14:35:25.204731 18327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:35:25.223517 18327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:35:25.227308 18327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:35:25.227527 18327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:35:25.228217 18327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:35:25.229079 18327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:35:25.229161 18327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:35:25.229228 18327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:35:25.229279 18327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:35:25.238575 18327 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.3:45385
I20260619 14:35:25.238649 18455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.3:45385 every 8 connection(s)
I20260619 14:35:25.239632 18327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
I20260619 14:35:25.248333 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 18327
Jun 19 14:35:25 dist-test-slave-70rm krb5kdc[17415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879725, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.3@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:35:25.254441 18188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:25.241526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.3:50467 (local address 127.2.2.62:41891)
0619 14:35:25.241699 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 14:35:25.241704 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:35:25.242568 (+ 864us) server_negotiation.cc:407] Connection header received
0619 14:35:25.243950 (+ 1382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:35:25.243953 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:35:25.244006 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:35:25.244098 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:35:25.245357 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:35:25.245926 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:35:25.247211 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:35:25.247431 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:35:25.247521 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:35:25.250546 (+ 3025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:35:25.250553 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:35:25.250559 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:35:25.250589 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:35:25.252207 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:35:25.252788 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:35:25.252791 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:35:25.252792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:35:25.252868 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:35:25.253266 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:35:25.253269 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:35:25.253271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:35:25.253516 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:35:25.253621 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 14:35:25.254161 (+ 540us) server_negotiation.cc:299] Negotiation successful
0619 14:35:25.254297 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 14:35:25.255535 18458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:25.241910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41891 (local address 127.2.2.3:50467)
0619 14:35:25.242408 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:25.242454 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 14:35:25.243701 (+ 1247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:35:25.244313 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:35:25.244321 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:35:25.244693 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:35:25.245198 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:35:25.245215 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:35:25.246063 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:35:25.246067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:35:25.247065 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:35:25.247074 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:35:25.247526 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:35:25.247593 (+ 67us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:35:25.247752 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:35:25.248484 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:35:25.248501 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:35:25.250384 (+ 1883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:35:25.252321 (+ 1937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:35:25.252325 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:35:25.252337 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:35:25.252682 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:35:25.252980 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:35:25.252983 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:35:25.252985 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:35:25.253149 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:35:25.253625 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:35:25.253630 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:35:25.253864 (+ 234us) client_negotiation.cc:769] Sending connection context
0619 14:35:25.254017 (+ 153us) client_negotiation.cc:240] Negotiation successful
0619 14:35:25.254413 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":164,"threads_started":1}
I20260619 14:35:25.256923 18456 heartbeater.cc:347] Connected to a master server at 127.2.2.62:41891
I20260619 14:35:25.257223 18456 heartbeater.cc:464] Registering TS with master...
I20260619 14:35:25.257753 18459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:25.249022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37760 (local address 127.2.2.62:41891)
0619 14:35:25.249321 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0619 14:35:25.249325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:35:25.249545 (+ 220us) server_negotiation.cc:407] Connection header received
0619 14:35:25.249629 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:35:25.249632 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:35:25.249692 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:35:25.249788 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:35:25.250600 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:35:25.251265 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:35:25.252498 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:35:25.252864 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:35:25.252969 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:35:25.254319 (+ 1350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:35:25.254323 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:35:25.254325 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:35:25.254350 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:35:25.256021 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:35:25.256514 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:35:25.256518 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:35:25.256520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:35:25.256610 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:35:25.256873 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:35:25.256876 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:35:25.256878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:35:25.257124 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:35:25.257324 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0619 14:35:25.257499 (+ 175us) server_negotiation.cc:299] Negotiation successful
0619 14:35:25.257597 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":125,"threads_started":1}
I20260619 14:35:25.257848 18456 heartbeater.cc:511] Master 127.2.2.62:41891 requested a full tablet report, sending...
I20260619 14:35:25.259093 17998 ts_manager.cc:194] Registered new tserver with Master: 8d1b5a5b42be462895944884b9163686 (127.2.2.3:45385)
I20260619 14:35:25.259706 17998 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.2.3@KRBTEST.COM'} at 127.2.2.3:50467
I20260619 14:35:25.271610 2056 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:35:25.275910 18459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:25.272337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37776 (local address 127.2.2.62:41891)
0619 14:35:25.272470 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 14:35:25.272474 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:35:25.272572 (+ 98us) server_negotiation.cc:407] Connection header received
0619 14:35:25.272737 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:35:25.272739 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:35:25.272835 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:35:25.272923 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:35:25.273133 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:35:25.273631 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:35:25.274665 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:35:25.274907 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:35:25.275006 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:35:25.275354 (+ 348us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:35:25.275497 (+ 143us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0619 14:35:25.275760 (+ 263us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37776: token expired
Metrics: {"server-negotiator.queue_time_us":56}
W20260619 14:35:25.276001 18459 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37776: token expired
W20260619 14:35:25.276352 17922 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.2.2.62:41891)
I20260619 14:35:25.283731 18459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:25.276759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37790 (local address 127.2.2.62:41891)
0619 14:35:25.276882 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 14:35:25.276885 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:35:25.276898 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:35:25.276988 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:35:25.276990 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:35:25.277026 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:35:25.277089 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:35:25.277718 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:35:25.278233 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:35:25.279184 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:35:25.279406 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:35:25.279480 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:35:25.280969 (+ 1489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:35:25.280972 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:35:25.280973 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:35:25.280990 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:35:25.282344 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:35:25.282728 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:35:25.282730 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:35:25.282732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:35:25.282803 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:35:25.283075 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:35:25.283077 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:35:25.283078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:35:25.283232 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:35:25.283327 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 14:35:25.283525 (+ 198us) server_negotiation.cc:299] Negotiation successful
0619 14:35:25.283618 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 14:35:25.286351 17998 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37790:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:35:25.289765 17998 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:35:25.297896 18120 tablet_service.cc:1467] Processing CreateTablet for tablet aadacf5a44c743d280964d0ae1e52f23 (DEFAULT_TABLE table=test-table [id=a6f0aedc3c29427cb214a586f9d9d71e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:35:25.298606 18120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aadacf5a44c743d280964d0ae1e52f23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:35:25.301517 18078 tablet_bootstrap.cc:492] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c: Bootstrap starting.
I20260619 14:35:25.302456 18078 tablet_bootstrap.cc:654] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c: Neither blocks nor log segments found. Creating new log.
I20260619 14:35:25.303158 18193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:25.296707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.2:33101 (local address 127.0.0.1:37086)
0619 14:35:25.296832 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:25.296848 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:35:25.296937 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:35:25.297839 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:35:25.297842 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:35:25.297859 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:35:25.297974 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:35:25.297981 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:35:25.299833 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:35:25.299837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:35:25.300972 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:35:25.300987 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:35:25.302716 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:35:25.302720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:35:25.302839 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:35:25.302856 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 14:35:25.302924 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 14:35:25.303030 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":41}
I20260619 14:35:25.303239 18465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:25.296916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37086 (local address 127.2.2.2:33101)
0619 14:35:25.297281 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0619 14:35:25.297295 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:35:25.297319 (+ 24us) server_negotiation.cc:407] Connection header received
0619 14:35:25.297396 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:35:25.297402 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:35:25.297662 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:35:25.297869 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:35:25.298138 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:35:25.299715 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:35:25.301258 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:35:25.302600 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:35:25.302739 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:35:25.302918 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0619 14:35:25.302975 (+ 57us) server_negotiation.cc:299] Negotiation successful
0619 14:35:25.303068 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":134,"threads_started":1}
I20260619 14:35:25.303306 18467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:25.297336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43182 (local address 127.2.2.3:45385)
0619 14:35:25.297761 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0619 14:35:25.297773 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:35:25.297795 (+ 22us) server_negotiation.cc:407] Connection header received
0619 14:35:25.297869 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:35:25.297886 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:35:25.298095 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:35:25.298272 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:35:25.298458 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:35:25.299501 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:35:25.300668 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:35:25.301343 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:35:25.302419 (+ 1076us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:35:25.302596 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 14:35:25.303058 (+ 462us) server_negotiation.cc:299] Negotiation successful
0619 14:35:25.303155 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":140,"threads_started":1}
I20260619 14:35:25.303856 18466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:25.296967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.3:45385 (local address 127.0.0.1:43182)
0619 14:35:25.297548 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:25.297565 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:35:25.297645 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:35:25.298263 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:35:25.298265 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:35:25.298283 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:35:25.298365 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:35:25.298372 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:35:25.299620 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:35:25.299624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:35:25.300541 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:35:25.300551 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:35:25.302653 (+ 2102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:35:25.302658 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:35:25.302754 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:35:25.302839 (+ 85us) client_negotiation.cc:769] Sending connection context
0619 14:35:25.302918 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 14:35:25.303761 (+ 843us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":784,"thread_start_us":98,"threads_started":1}
I20260619 14:35:25.304868 18078 tablet_bootstrap.cc:492] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c: No bootstrap required, opened a new log
I20260619 14:35:25.304956 18078 ts_tablet_manager.cc:1397] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 14:35:25.305387 18078 raft_consensus.cc:348] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ea989b097c44838b4491edae66ac67" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 33101 } } peers { permanent_uuid: "8d1b5a5b42be462895944884b9163686" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45385 } } peers { permanent_uuid: "ed48fbe415dd43babd6582e39e83e02c" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 36581 } }
I20260619 14:35:25.305495 18078 raft_consensus.cc:374] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:35:25.305529 18078 raft_consensus.cc:729] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed48fbe415dd43babd6582e39e83e02c, State: Initialized, Role: FOLLOWER
I20260619 14:35:25.305658 18078 consensus_queue.cc:260] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ea989b097c44838b4491edae66ac67" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 33101 } } peers { permanent_uuid: "8d1b5a5b42be462895944884b9163686" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45385 } } peers { permanent_uuid: "ed48fbe415dd43babd6582e39e83e02c" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 36581 } }
I20260619 14:35:25.305913 18078 ts_tablet_manager.cc:1428] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 14:35:25.306185 18258 tablet_service.cc:1467] Processing CreateTablet for tablet aadacf5a44c743d280964d0ae1e52f23 (DEFAULT_TABLE table=test-table [id=a6f0aedc3c29427cb214a586f9d9d71e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:35:25.306739 18390 tablet_service.cc:1467] Processing CreateTablet for tablet aadacf5a44c743d280964d0ae1e52f23 (DEFAULT_TABLE table=test-table [id=a6f0aedc3c29427cb214a586f9d9d71e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:35:25.307133 18258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aadacf5a44c743d280964d0ae1e52f23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:35:25.307626 18390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aadacf5a44c743d280964d0ae1e52f23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:35:25.313892 18469 tablet_bootstrap.cc:492] T aadacf5a44c743d280964d0ae1e52f23 P 37ea989b097c44838b4491edae66ac67: Bootstrap starting.
I20260619 14:35:25.314024 18470 tablet_bootstrap.cc:492] T aadacf5a44c743d280964d0ae1e52f23 P 8d1b5a5b42be462895944884b9163686: Bootstrap starting.
I20260619 14:35:25.316540 18470 tablet_bootstrap.cc:654] T aadacf5a44c743d280964d0ae1e52f23 P 8d1b5a5b42be462895944884b9163686: Neither blocks nor log segments found. Creating new log.
I20260619 14:35:25.316709 18469 tablet_bootstrap.cc:654] T aadacf5a44c743d280964d0ae1e52f23 P 37ea989b097c44838b4491edae66ac67: Neither blocks nor log segments found. Creating new log.
I20260619 14:35:25.317634 18470 log.cc:824] T aadacf5a44c743d280964d0ae1e52f23 P 8d1b5a5b42be462895944884b9163686: Log is configured to *not* fsync() on all Append() calls
I20260619 14:35:25.317646 18469 log.cc:824] T aadacf5a44c743d280964d0ae1e52f23 P 37ea989b097c44838b4491edae66ac67: Log is configured to *not* fsync() on all Append() calls
I20260619 14:35:25.319430 18469 tablet_bootstrap.cc:492] T aadacf5a44c743d280964d0ae1e52f23 P 37ea989b097c44838b4491edae66ac67: No bootstrap required, opened a new log
I20260619 14:35:25.319444 18470 tablet_bootstrap.cc:492] T aadacf5a44c743d280964d0ae1e52f23 P 8d1b5a5b42be462895944884b9163686: No bootstrap required, opened a new log
I20260619 14:35:25.319635 18469 ts_tablet_manager.cc:1397] T aadacf5a44c743d280964d0ae1e52f23 P 37ea989b097c44838b4491edae66ac67: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260619 14:35:25.319675 18470 ts_tablet_manager.cc:1397] T aadacf5a44c743d280964d0ae1e52f23 P 8d1b5a5b42be462895944884b9163686: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 14:35:25.322685 18469 raft_consensus.cc:348] T aadacf5a44c743d280964d0ae1e52f23 P 37ea989b097c44838b4491edae66ac67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ea989b097c44838b4491edae66ac67" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 33101 } } peers { permanent_uuid: "8d1b5a5b42be462895944884b9163686" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45385 } } peers { permanent_uuid: "ed48fbe415dd43babd6582e39e83e02c" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 36581 } }
I20260619 14:35:25.322685 18470 raft_consensus.cc:348] T aadacf5a44c743d280964d0ae1e52f23 P 8d1b5a5b42be462895944884b9163686 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ea989b097c44838b4491edae66ac67" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 33101 } } peers { permanent_uuid: "8d1b5a5b42be462895944884b9163686" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45385 } } peers { permanent_uuid: "ed48fbe415dd43babd6582e39e83e02c" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 36581 } }
I20260619 14:35:25.322934 18469 raft_consensus.cc:374] T aadacf5a44c743d280964d0ae1e52f23 P 37ea989b097c44838b4491edae66ac67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:35:25.322929 18470 raft_consensus.cc:374] T aadacf5a44c743d280964d0ae1e52f23 P 8d1b5a5b42be462895944884b9163686 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:35:25.322993 18469 raft_consensus.cc:729] T aadacf5a44c743d280964d0ae1e52f23 P 37ea989b097c44838b4491edae66ac67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37ea989b097c44838b4491edae66ac67, State: Initialized, Role: FOLLOWER
I20260619 14:35:25.323012 18470 raft_consensus.cc:729] T aadacf5a44c743d280964d0ae1e52f23 P 8d1b5a5b42be462895944884b9163686 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d1b5a5b42be462895944884b9163686, State: Initialized, Role: FOLLOWER
I20260619 14:35:25.323390 18469 consensus_queue.cc:260] T aadacf5a44c743d280964d0ae1e52f23 P 37ea989b097c44838b4491edae66ac67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ea989b097c44838b4491edae66ac67" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 33101 } } peers { permanent_uuid: "8d1b5a5b42be462895944884b9163686" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45385 } } peers { permanent_uuid: "ed48fbe415dd43babd6582e39e83e02c" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 36581 } }
I20260619 14:35:25.323410 18470 consensus_queue.cc:260] T aadacf5a44c743d280964d0ae1e52f23 P 8d1b5a5b42be462895944884b9163686 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ea989b097c44838b4491edae66ac67" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 33101 } } peers { permanent_uuid: "8d1b5a5b42be462895944884b9163686" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45385 } } peers { permanent_uuid: "ed48fbe415dd43babd6582e39e83e02c" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 36581 } }
I20260619 14:35:25.324008 18324 heartbeater.cc:503] Master 127.2.2.62:41891 was elected leader, sending a full tablet report...
I20260619 14:35:25.324036 18456 heartbeater.cc:503] Master 127.2.2.62:41891 was elected leader, sending a full tablet report...
I20260619 14:35:25.324384 18470 ts_tablet_manager.cc:1428] T aadacf5a44c743d280964d0ae1e52f23 P 8d1b5a5b42be462895944884b9163686: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 14:35:25.325353 18469 ts_tablet_manager.cc:1428] T aadacf5a44c743d280964d0ae1e52f23 P 37ea989b097c44838b4491edae66ac67: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
W20260619 14:35:25.356379 18187 tablet.cc:2369] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:35:25.490656 18457 tablet.cc:2369] T aadacf5a44c743d280964d0ae1e52f23 P 8d1b5a5b42be462895944884b9163686: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:35:25.509825 18192 raft_consensus.cc:482] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:35:25.510104 18192 raft_consensus.cc:504] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ea989b097c44838b4491edae66ac67" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 33101 } } peers { permanent_uuid: "8d1b5a5b42be462895944884b9163686" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45385 } } peers { permanent_uuid: "ed48fbe415dd43babd6582e39e83e02c" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 36581 } }
I20260619 14:35:25.511389 18192 leader_election.cc:288] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37ea989b097c44838b4491edae66ac67 (127.2.2.2:33101), 8d1b5a5b42be462895944884b9163686 (127.2.2.3:45385)
I20260619 14:35:25.515853 18467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:25.511790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:51733 (local address 127.2.2.3:45385)
0619 14:35:25.511909 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 14:35:25.511914 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:35:25.512126 (+ 212us) server_negotiation.cc:407] Connection header received
0619 14:35:25.512286 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:35:25.512289 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:35:25.512347 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:35:25.512430 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:35:25.512835 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:35:25.513364 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:35:25.514653 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:35:25.515325 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:35:25.515427 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:35:25.515484 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 14:35:25.515680 (+ 196us) server_negotiation.cc:299] Negotiation successful
0619 14:35:25.515756 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 14:35:25.515851 18465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:25.511772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:49307 (local address 127.2.2.2:33101)
0619 14:35:25.511909 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 14:35:25.511914 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:35:25.512306 (+ 392us) server_negotiation.cc:407] Connection header received
0619 14:35:25.512388 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:35:25.512509 (+ 121us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:35:25.512569 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:35:25.512653 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:35:25.512911 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:35:25.513387 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:35:25.514729 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:35:25.515224 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:35:25.515308 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:35:25.515362 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 14:35:25.515634 (+ 272us) server_negotiation.cc:299] Negotiation successful
0619 14:35:25.515715 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 14:35:25.515870 18475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:25.511657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.2:33101 (local address 127.2.2.1:49307)
0619 14:35:25.512159 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:25.512175 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:35:25.512303 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:35:25.512671 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:35:25.512674 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:35:25.512689 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:35:25.512765 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:35:25.512770 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:35:25.513520 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:35:25.513523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:35:25.514583 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:35:25.514613 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:35:25.515343 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:35:25.515349 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:35:25.515446 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:35:25.515498 (+ 52us) client_negotiation.cc:769] Sending connection context
0619 14:35:25.515588 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 14:35:25.515687 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"mutex_wait_us":106,"thread_start_us":73,"threads_started":1}
I20260619 14:35:25.515877 18476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:25.511656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.3:45385 (local address 127.2.2.1:51733)
0619 14:35:25.512033 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:25.512046 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:35:25.512157 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:35:25.512548 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:35:25.512551 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:35:25.512569 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:35:25.512713 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:35:25.512722 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:35:25.513480 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:35:25.513485 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:35:25.514531 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:35:25.514541 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:35:25.515462 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:35:25.515464 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:35:25.515537 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:35:25.515554 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 14:35:25.515650 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 14:35:25.515715 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":74,"thread_start_us":89,"threads_started":1}
I20260619 14:35:25.516566 18278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aadacf5a44c743d280964d0ae1e52f23" candidate_uuid: "ed48fbe415dd43babd6582e39e83e02c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37ea989b097c44838b4491edae66ac67" is_pre_election: true
I20260619 14:35:25.516916 18278 raft_consensus.cc:2454] T aadacf5a44c743d280964d0ae1e52f23 P 37ea989b097c44838b4491edae66ac67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed48fbe415dd43babd6582e39e83e02c in term 0.
I20260619 14:35:25.516958 18410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aadacf5a44c743d280964d0ae1e52f23" candidate_uuid: "ed48fbe415dd43babd6582e39e83e02c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d1b5a5b42be462895944884b9163686" is_pre_election: true
I20260619 14:35:25.517292 18410 raft_consensus.cc:2454] T aadacf5a44c743d280964d0ae1e52f23 P 8d1b5a5b42be462895944884b9163686 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed48fbe415dd43babd6582e39e83e02c in term 0.
I20260619 14:35:25.517488 18073 leader_election.cc:302] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37ea989b097c44838b4491edae66ac67, ed48fbe415dd43babd6582e39e83e02c; no voters:
I20260619 14:35:25.517796 18192 raft_consensus.cc:2781] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:35:25.517896 18192 raft_consensus.cc:482] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:35:25.517951 18192 raft_consensus.cc:3037] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:35:25.519018 18192 raft_consensus.cc:504] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ea989b097c44838b4491edae66ac67" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 33101 } } peers { permanent_uuid: "8d1b5a5b42be462895944884b9163686" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45385 } } peers { permanent_uuid: "ed48fbe415dd43babd6582e39e83e02c" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 36581 } }
I20260619 14:35:25.519419 18192 leader_election.cc:288] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c [CANDIDATE]: Term 1 election: Requested vote from peers 37ea989b097c44838b4491edae66ac67 (127.2.2.2:33101), 8d1b5a5b42be462895944884b9163686 (127.2.2.3:45385)
I20260619 14:35:25.519850 18278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aadacf5a44c743d280964d0ae1e52f23" candidate_uuid: "ed48fbe415dd43babd6582e39e83e02c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37ea989b097c44838b4491edae66ac67"
I20260619 14:35:25.519958 18410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aadacf5a44c743d280964d0ae1e52f23" candidate_uuid: "ed48fbe415dd43babd6582e39e83e02c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d1b5a5b42be462895944884b9163686"
I20260619 14:35:25.520001 18278 raft_consensus.cc:3037] T aadacf5a44c743d280964d0ae1e52f23 P 37ea989b097c44838b4491edae66ac67 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:35:25.520098 18410 raft_consensus.cc:3037] T aadacf5a44c743d280964d0ae1e52f23 P 8d1b5a5b42be462895944884b9163686 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:35:25.521132 18410 raft_consensus.cc:2454] T aadacf5a44c743d280964d0ae1e52f23 P 8d1b5a5b42be462895944884b9163686 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed48fbe415dd43babd6582e39e83e02c in term 1.
I20260619 14:35:25.521289 18278 raft_consensus.cc:2454] T aadacf5a44c743d280964d0ae1e52f23 P 37ea989b097c44838b4491edae66ac67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed48fbe415dd43babd6582e39e83e02c in term 1.
I20260619 14:35:25.521507 18075 leader_election.cc:302] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d1b5a5b42be462895944884b9163686, ed48fbe415dd43babd6582e39e83e02c; no voters:
I20260619 14:35:25.521695 18192 raft_consensus.cc:2781] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:35:25.521895 18192 raft_consensus.cc:686] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c [term 1 LEADER]: Becoming Leader. State: Replica: ed48fbe415dd43babd6582e39e83e02c, State: Running, Role: LEADER
I20260619 14:35:25.522226 18192 consensus_queue.cc:237] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ea989b097c44838b4491edae66ac67" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 33101 } } peers { permanent_uuid: "8d1b5a5b42be462895944884b9163686" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45385 } } peers { permanent_uuid: "ed48fbe415dd43babd6582e39e83e02c" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 36581 } }
I20260619 14:35:25.525481 17998 catalog_manager.cc:5495] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c reported cstate change: term changed from 0 to 1, leader changed from <none> to ed48fbe415dd43babd6582e39e83e02c (127.2.2.1). New cstate: current_term: 1 leader_uuid: "ed48fbe415dd43babd6582e39e83e02c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ea989b097c44838b4491edae66ac67" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 33101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d1b5a5b42be462895944884b9163686" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed48fbe415dd43babd6582e39e83e02c" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 36581 } health_report { overall_health: HEALTHY } } }
I20260619 14:35:25.539151 18479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:25.535484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38356 (local address 127.2.2.1:36581)
0619 14:35:25.535817 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0619 14:35:25.535824 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:35:25.535837 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:35:25.535914 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:35:25.535916 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:35:25.535969 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:35:25.536077 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:35:25.536288 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:35:25.536813 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:35:25.537642 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:35:25.537874 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:35:25.537957 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:35:25.538251 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:35:25.538699 (+ 448us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:35:25.538792 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 14:35:25.538876 (+ 84us) server_negotiation.cc:299] Negotiation successful
0619 14:35:25.538927 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":149,"threads_started":1}
I20260619 14:35:25.543097 18278 raft_consensus.cc:1261] T aadacf5a44c743d280964d0ae1e52f23 P 37ea989b097c44838b4491edae66ac67 [term 1 FOLLOWER]: Refusing update from remote peer ed48fbe415dd43babd6582e39e83e02c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:35:25.543097 18410 raft_consensus.cc:1261] T aadacf5a44c743d280964d0ae1e52f23 P 8d1b5a5b42be462895944884b9163686 [term 1 FOLLOWER]: Refusing update from remote peer ed48fbe415dd43babd6582e39e83e02c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:35:25.544040 18192 consensus_queue.cc:1035] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c [LEADER]: Connected to new peer: Peer: permanent_uuid: "37ea989b097c44838b4491edae66ac67" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 33101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:35:25.544288 18477 consensus_queue.cc:1035] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d1b5a5b42be462895944884b9163686" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:35:25.559860 17997 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:35:25.560056 17997 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:35:25.562533 17997 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f: Sending DeleteTablet for 3 replicas of tablet aadacf5a44c743d280964d0ae1e52f23
I20260619 14:35:25.563320 18120 tablet_service.cc:1514] Processing DeleteTablet for tablet aadacf5a44c743d280964d0ae1e52f23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:35:25 UTC) from {username='kudu', principal='kudu/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:38340
I20260619 14:35:25.563496 18390 tablet_service.cc:1514] Processing DeleteTablet for tablet aadacf5a44c743d280964d0ae1e52f23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:35:25 UTC) from {username='kudu', principal='kudu/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:43182
I20260619 14:35:25.563495 18258 tablet_service.cc:1514] Processing DeleteTablet for tablet aadacf5a44c743d280964d0ae1e52f23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:35:25 UTC) from {username='kudu', principal='kudu/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:37086
I20260619 14:35:25.563755 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 18036
I20260619 14:35:25.565457 18492 tablet_replica.cc:330] T aadacf5a44c743d280964d0ae1e52f23 P 8d1b5a5b42be462895944884b9163686: stopping tablet replica
I20260619 14:35:25.565835 18492 raft_consensus.cc:2229] T aadacf5a44c743d280964d0ae1e52f23 P 8d1b5a5b42be462895944884b9163686 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:35:25.566205 18492 raft_consensus.cc:2258] T aadacf5a44c743d280964d0ae1e52f23 P 8d1b5a5b42be462895944884b9163686 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:35:25.566403 18491 tablet_replica.cc:330] T aadacf5a44c743d280964d0ae1e52f23 P 37ea989b097c44838b4491edae66ac67: stopping tablet replica
I20260619 14:35:25.566720 18491 raft_consensus.cc:2229] T aadacf5a44c743d280964d0ae1e52f23 P 37ea989b097c44838b4491edae66ac67 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:35:25.567047 18491 raft_consensus.cc:2258] T aadacf5a44c743d280964d0ae1e52f23 P 37ea989b097c44838b4491edae66ac67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:35:25.567873 18492 ts_tablet_manager.cc:1905] T aadacf5a44c743d280964d0ae1e52f23 P 8d1b5a5b42be462895944884b9163686: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:35:25.568244 18491 ts_tablet_manager.cc:1905] T aadacf5a44c743d280964d0ae1e52f23 P 37ea989b097c44838b4491edae66ac67: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:35:25.570393 18491 ts_tablet_manager.cc:1918] T aadacf5a44c743d280964d0ae1e52f23 P 37ea989b097c44838b4491edae66ac67: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:35:25.570505 18491 log.cc:1196] T aadacf5a44c743d280964d0ae1e52f23 P 37ea989b097c44838b4491edae66ac67: Deleting WAL directory at /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-1/wal/wals/aadacf5a44c743d280964d0ae1e52f23
I20260619 14:35:25.570787 18491 ts_tablet_manager.cc:1939] T aadacf5a44c743d280964d0ae1e52f23 P 37ea989b097c44838b4491edae66ac67: Deleting consensus metadata
I20260619 14:35:25.571363 18492 ts_tablet_manager.cc:1918] T aadacf5a44c743d280964d0ae1e52f23 P 8d1b5a5b42be462895944884b9163686: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:35:25.571463 18492 log.cc:1196] T aadacf5a44c743d280964d0ae1e52f23 P 8d1b5a5b42be462895944884b9163686: Deleting WAL directory at /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-2/wal/wals/aadacf5a44c743d280964d0ae1e52f23
W20260619 14:35:25.571784 17982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.2.1:36581 (error 108)
I20260619 14:35:25.572010 18492 ts_tablet_manager.cc:1939] T aadacf5a44c743d280964d0ae1e52f23 P 8d1b5a5b42be462895944884b9163686: Deleting consensus metadata
I20260619 14:35:25.572448 17983 catalog_manager.cc:4841] TS 37ea989b097c44838b4491edae66ac67 (127.2.2.2:33101): tablet aadacf5a44c743d280964d0ae1e52f23 (table test-table [id=a6f0aedc3c29427cb214a586f9d9d71e]) successfully deleted
I20260619 14:35:25.573179 17985 catalog_manager.cc:4841] TS 8d1b5a5b42be462895944884b9163686 (127.2.2.3:45385): tablet aadacf5a44c743d280964d0ae1e52f23 (table test-table [id=a6f0aedc3c29427cb214a586f9d9d71e]) successfully deleted
W20260619 14:35:25.573237 18466 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:35:25.572575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.1:36581 (local address 127.0.0.1:38366)
0619 14:35:25.572746 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:25.572763 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:35:25.572848 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:35:25.573130 (+ 282us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.1:36581: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":97}
I20260619 14:35:25.573375 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 18191
W20260619 14:35:25.573407 17982 catalog_manager.cc:4584] TS ed48fbe415dd43babd6582e39e83e02c (127.2.2.1:36581): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet aadacf5a44c743d280964d0ae1e52f23: Network error: Client connection negotiation failed: client connection to 127.2.2.1:36581: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 14:35:25.579494 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 18327
I20260619 14:35:25.586683 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 17965
I20260619 14:35:45.594575 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:41891
--webserver_interface=127.2.2.62
--webserver_port=34103
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:37921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.2.62:41891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:35:45.712424 18494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:35:45.712707 18494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:35:45.712749 18494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:35:45.716897 18494 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 14:35:45.716949 18494 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 14:35:45.716964 18494 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 14:35:45.716979 18494 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:35:45.716998 18494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:35:45.717064 18494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:35:45.717085 18494 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:35:45.717108 18494 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:35:45.722707 18494 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:37921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.2.62:41891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:41891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.2.62
--webserver_port=34103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:35:45.723857 18494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:35:45.724954 18494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:35:45.734409 18500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:35:45.734382 18499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:35:45.734371 18502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:35:45.734478 18494 server_base.cc:1034] running on GCE node
I20260619 14:35:45.735312 18494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:35:45.736586 18494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:35:45.737795 18494 hybrid_clock.cc:648] HybridClock initialized: now 1781879745737779 us; error 38 us; skew 500 ppm
Jun 19 14:35:45 dist-test-slave-70rm krb5kdc[17415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879745, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:35:45.744987 18494 init.cc:377] Logged in from keytab as kudu/127.2.2.62@KRBTEST.COM (short username kudu)
I20260619 14:35:45.746109 18494 webserver.cc:460] Webserver started at http://127.2.2.62:34103/ using document root <none> and password file <none>
I20260619 14:35:45.746793 18494 fs_manager.cc:359] Metadata directory not provided
I20260619 14:35:45.746898 18494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:35:45.751164 18494 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260619 14:35:45.753366 18509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:35:45.754719 18494 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 14:35:45.754900 18494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/master-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/master-0/wal
uuid: "b57bdd13c44f4b00a3d67fc5b168cb3f"
format_stamp: "Formatted at 2026-06-19 14:35:03 on dist-test-slave-70rm"
server_key: "536d4638e100092d76e2479fb40d531e"
server_key_iv: "b9de7b5fc7fe0ee9164acb5b1e8cea70"
server_key_version: "encryptionkey@0"
I20260619 14:35:45.755489 18494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:35:45.769141 18494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:35:45.772138 18494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:35:45.772351 18494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:35:45.779767 18494 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.62:41891
I20260619 14:35:45.780081 18561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.62:41891 every 8 connection(s)
I20260619 14:35:45.780934 18494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
I20260619 14:35:45.781555 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 18494
I20260619 14:35:45.781996 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.1:36581
--local_ip_for_outbound_sockets=127.2.2.1
--tserver_master_addrs=127.2.2.62:41891
--webserver_port=43115
--webserver_interface=127.2.2.1
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab
--principal=kudu/127.2.2.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:37921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:35:45.785920 18562 sys_catalog.cc:263] Verifying existing consensus state
I20260619 14:35:45.788659 18562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f: Bootstrap starting.
I20260619 14:35:45.813000 18562 log.cc:824] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f: Log is configured to *not* fsync() on all Append() calls
I20260619 14:35:45.826895 18562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:35:45.827384 18562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f: Bootstrap complete.
I20260619 14:35:45.833046 18562 raft_consensus.cc:348] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57bdd13c44f4b00a3d67fc5b168cb3f" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41891 } }
I20260619 14:35:45.833583 18562 raft_consensus.cc:729] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b57bdd13c44f4b00a3d67fc5b168cb3f, State: Initialized, Role: FOLLOWER
I20260619 14:35:45.834203 18562 consensus_queue.cc:260] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57bdd13c44f4b00a3d67fc5b168cb3f" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41891 } }
I20260619 14:35:45.834385 18562 raft_consensus.cc:388] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:35:45.834462 18562 raft_consensus.cc:482] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:35:45.834566 18562 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f [term 2 FOLLOWER]: Advancing to term 3
I20260619 14:35:45.837186 18562 raft_consensus.cc:504] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57bdd13c44f4b00a3d67fc5b168cb3f" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41891 } }
I20260619 14:35:45.837579 18562 leader_election.cc:302] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b57bdd13c44f4b00a3d67fc5b168cb3f; no voters:
I20260619 14:35:45.837934 18562 leader_election.cc:288] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 14:35:45.838166 18566 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f [term 3 FOLLOWER]: Leader election won for term 3
I20260619 14:35:45.838512 18566 raft_consensus.cc:686] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f [term 3 LEADER]: Becoming Leader. State: Replica: b57bdd13c44f4b00a3d67fc5b168cb3f, State: Running, Role: LEADER
I20260619 14:35:45.839007 18566 consensus_queue.cc:237] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57bdd13c44f4b00a3d67fc5b168cb3f" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41891 } }
I20260619 14:35:45.839316 18562 sys_catalog.cc:564] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:35:45.840627 18567 sys_catalog.cc:455] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "b57bdd13c44f4b00a3d67fc5b168cb3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57bdd13c44f4b00a3d67fc5b168cb3f" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41891 } } }
I20260619 14:35:45.840627 18568 sys_catalog.cc:455] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b57bdd13c44f4b00a3d67fc5b168cb3f. Latest consensus state: current_term: 3 leader_uuid: "b57bdd13c44f4b00a3d67fc5b168cb3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57bdd13c44f4b00a3d67fc5b168cb3f" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41891 } } }
I20260619 14:35:45.840785 18567 sys_catalog.cc:458] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f [sys.catalog]: This master's current role is: LEADER
I20260619 14:35:45.840794 18568 sys_catalog.cc:458] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f [sys.catalog]: This master's current role is: LEADER
I20260619 14:35:45.844764 18581 catalog_manager.cc:1259] Loaded cluster ID: f112e01fc9464ef8b0580b565920bf8a
I20260619 14:35:45.844846 18581 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f: loading cluster ID for follower catalog manager: success
I20260619 14:35:45.847424 18581 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b57bdd13c44f4b00a3d67fc5b168cb3f: acquiring CA information for follower catalog manager: success
F20260619 14:35:45.848681 18581 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f5f2811e9ad google::LogMessage::Fail() at ??:0
@ 0x7f5f2812188a google::LogMessage::SendToLog() at ??:0
@ 0x7f5f2811e4de google::LogMessage::Flush() at ??:0
@ 0x7f5f2811fa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f5f2ade218d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f5f32a83989 kudu::master::CatalogManager::LoadTspkEntries() at ??:0
@ 0x7f5f32a60023 kudu::master::CatalogManager::PrepareFollowerTokenVerifier() at ??:0
@ 0x7f5f32a6054b kudu::master::CatalogManager::PrepareFollower() at ??:0
@ 0x7f5f32a59652 kudu::master::CatalogManagerBgTasks::Run() at ??:0
@ 0x7f5f32a589e9 _ZZN4kudu6master21CatalogManagerBgTasks4InitEvENKUlvE_clEv at ??:0
@ 0x7f5f32a9f541 _ZNSt17_Function_handlerIFvvEZN4kudu6master21CatalogManagerBgTasks4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f5f2a361512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f5f2b9a56db start_thread at ??:0
@ 0x7f5f26c1c61f clone at ??:0
W20260619 14:35:45.945364 18564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:35:45.945775 18564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:35:45.945856 18564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:35:45.952399 18564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:35:45.952507 18564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:35:45.952605 18564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.1
I20260619 14:35:45.960837 18564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:37921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab
--principal=kudu/127.2.2.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.1:36581
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.2.1
--webserver_port=43115
--tserver_master_addrs=127.2.2.62:41891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.1
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:35:45.962268 18564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:35:45.963673 18564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:35:45.975303 18602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:35:45.975297 18603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:35:45.975476 18564 server_base.cc:1034] running on GCE node
W20260619 14:35:45.975672 18605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:35:45.976173 18564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:35:45.977048 18564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:35:45.978225 18564 hybrid_clock.cc:648] HybridClock initialized: now 1781879745978188 us; error 46 us; skew 500 ppm
Jun 19 14:35:45 dist-test-slave-70rm krb5kdc[17415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879745, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:35:45.981796 18564 init.cc:377] Logged in from keytab as kudu/127.2.2.1@KRBTEST.COM (short username kudu)
I20260619 14:35:45.982833 18564 webserver.cc:460] Webserver started at http://127.2.2.1:43115/ using document root <none> and password file <none>
I20260619 14:35:45.983424 18564 fs_manager.cc:359] Metadata directory not provided
I20260619 14:35:45.983492 18564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:35:45.987823 18564 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:35:45.990320 18612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:35:45.991740 18564 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:35:45.991909 18564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-0/wal
uuid: "ed48fbe415dd43babd6582e39e83e02c"
format_stamp: "Formatted at 2026-06-19 14:35:03 on dist-test-slave-70rm"
server_key: "063a84231a70ecfe86ea55c3b4623636"
server_key_iv: "f52d9d4620e8c4aacadf7d13af1f6ed4"
server_key_version: "encryptionkey@0"
I20260619 14:35:45.992479 18564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:35:46.024427 18564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:35:46.028100 18564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:35:46.028338 18564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:35:46.028993 18564 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:35:46.030215 18619 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 14:35:46.035982 18564 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 14:35:46.036106 18564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260619 14:35:46.036196 18564 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 14:35:46.038111 18564 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 14:35:46.038249 18564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260619 14:35:46.038515 18619 tablet_bootstrap.cc:492] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c: Bootstrap starting.
I20260619 14:35:46.048980 18564 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.1:36581
I20260619 14:35:46.049269 18726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.1:36581 every 8 connection(s)
I20260619 14:35:46.050057 18564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
I20260619 14:35:46.051940 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 18564
I20260619 14:35:46.052379 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.2:33101
--local_ip_for_outbound_sockets=127.2.2.2
--tserver_master_addrs=127.2.2.62:41891
--webserver_port=40855
--webserver_interface=127.2.2.2
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab
--principal=kudu/127.2.2.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:37921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:35:46.060810 18619 log.cc:824] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c: Log is configured to *not* fsync() on all Append() calls
I20260619 14:35:46.068033 18619 tablet_bootstrap.cc:492] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:35:46.068444 18619 tablet_bootstrap.cc:492] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c: Bootstrap complete.
I20260619 14:35:46.069591 18619 ts_tablet_manager.cc:1397] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c: Time spent bootstrapping tablet: real 0.031s user 0.018s sys 0.009s
I20260619 14:35:46.072171 18619 raft_consensus.cc:348] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ea989b097c44838b4491edae66ac67" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 33101 } } peers { permanent_uuid: "8d1b5a5b42be462895944884b9163686" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45385 } } peers { permanent_uuid: "ed48fbe415dd43babd6582e39e83e02c" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 36581 } }
I20260619 14:35:46.072641 18619 raft_consensus.cc:729] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed48fbe415dd43babd6582e39e83e02c, State: Initialized, Role: FOLLOWER
I20260619 14:35:46.073158 18619 consensus_queue.cc:260] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ea989b097c44838b4491edae66ac67" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 33101 } } peers { permanent_uuid: "8d1b5a5b42be462895944884b9163686" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45385 } } peers { permanent_uuid: "ed48fbe415dd43babd6582e39e83e02c" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 36581 } }
I20260619 14:35:46.074236 18619 ts_tablet_manager.cc:1428] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
W20260619 14:35:46.092490 18729 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:35:46.052304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41891 (local address 127.2.2.1:36139)
0619 14:35:46.053700 (+ 1396us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:46.053769 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0619 14:35:46.054695 (+ 926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:35:46.091722 (+ 37027us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:41891: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1139,"thread_start_us":155,"threads_started":1}
W20260619 14:35:46.093042 18727 heartbeater.cc:644] Failed to heartbeat to 127.2.2.62:41891 (0 consecutive failures): Network error: Failed to ping master at 127.2.2.62:41891: Client connection negotiation failed: client connection to 127.2.2.62:41891: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 14:35:46.093766 18729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:46.093515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41891 (local address 127.2.2.1:40113)
0619 14:35:46.093639 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:46.093713 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:41891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 14:35:46.094462 18729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:46.094260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41891 (local address 127.2.2.1:35575)
0619 14:35:46.094360 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:46.094405 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:41891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 14:35:46.094614 18727 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 14:35:46.181309 18730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:35:46.181676 18730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:35:46.181756 18730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:35:46.185990 18730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:35:46.186110 18730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:35:46.186241 18730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.2
I20260619 14:35:46.192040 18730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:37921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab
--principal=kudu/127.2.2.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.2:33101
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.2.2
--webserver_port=40855
--tserver_master_addrs=127.2.2.62:41891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.2
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:35:46.193233 18730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:35:46.194355 18730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:35:46.204316 18737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:35:46.204274 18738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:35:46.204442 18730 server_base.cc:1034] running on GCE node
W20260619 14:35:46.204270 18740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:35:46.205013 18730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:35:46.205690 18730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:35:46.206853 18730 hybrid_clock.cc:648] HybridClock initialized: now 1781879746206834 us; error 32 us; skew 500 ppm
Jun 19 14:35:46 dist-test-slave-70rm krb5kdc[17415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879746, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:35:46.209738 18730 init.cc:377] Logged in from keytab as kudu/127.2.2.2@KRBTEST.COM (short username kudu)
I20260619 14:35:46.210800 18730 webserver.cc:460] Webserver started at http://127.2.2.2:40855/ using document root <none> and password file <none>
I20260619 14:35:46.211405 18730 fs_manager.cc:359] Metadata directory not provided
I20260619 14:35:46.211470 18730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:35:46.215696 18730 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:35:46.218253 18747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:35:46.219614 18730 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:35:46.219767 18730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-1/wal
uuid: "37ea989b097c44838b4491edae66ac67"
format_stamp: "Formatted at 2026-06-19 14:35:03 on dist-test-slave-70rm"
server_key: "821e7ea015dc5edf0313fdc50aa93224"
server_key_iv: "80baa16d283bb6b366df3cf486b93e40"
server_key_version: "encryptionkey@0"
I20260619 14:35:46.220373 18730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:35:46.241356 18730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:35:46.244722 18730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:35:46.244961 18730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:35:46.245605 18730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:35:46.246490 18730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:35:46.246562 18730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:35:46.246634 18730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:35:46.246677 18730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:35:46.255975 18730 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.2:33101
I20260619 14:35:46.255995 18860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.2:33101 every 8 connection(s)
I20260619 14:35:46.257134 18730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
I20260619 14:35:46.261658 18863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:46.259703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41891 (local address 127.2.2.2:59383)
0619 14:35:46.260274 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:46.260406 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:41891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":126,"threads_started":1}
W20260619 14:35:46.262395 18861 heartbeater.cc:644] Failed to heartbeat to 127.2.2.62:41891 (0 consecutive failures): Network error: Failed to ping master at 127.2.2.62:41891: Client connection negotiation failed: client connection to 127.2.2.62:41891: connect: Connection refused (error 111)
I20260619 14:35:46.263204 18863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:46.262931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41891 (local address 127.2.2.2:37587)
0619 14:35:46.263076 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:46.263143 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:41891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260619 14:35:46.263497 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 18730
I20260619 14:35:46.263959 18863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:46.263753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41891 (local address 127.2.2.2:55333)
0619 14:35:46.263861 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:46.263906 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:41891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 14:35:46.263999 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.3:45385
--local_ip_for_outbound_sockets=127.2.2.3
--tserver_master_addrs=127.2.2.62:41891
--webserver_port=44157
--webserver_interface=127.2.2.3
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab
--principal=kudu/127.2.2.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:37921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:35:46.264305 18861 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 14:35:46.381462 18864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:35:46.381767 18864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:35:46.381814 18864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:35:46.386240 18864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:35:46.386324 18864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:35:46.386422 18864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.3
I20260619 14:35:46.392236 18864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:37921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab
--principal=kudu/127.2.2.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.3:45385
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.2.3
--webserver_port=44157
--tserver_master_addrs=127.2.2.62:41891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.3
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:35:46.393417 18864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:35:46.394610 18864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:35:46.405016 18869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:35:46.405020 18872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:35:46.405128 18864 server_base.cc:1034] running on GCE node
W20260619 14:35:46.405016 18870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:35:46.405696 18864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:35:46.406404 18864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:35:46.407567 18864 hybrid_clock.cc:648] HybridClock initialized: now 1781879746407549 us; error 31 us; skew 500 ppm
Jun 19 14:35:46 dist-test-slave-70rm krb5kdc[17415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879746, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:35:46.410377 18864 init.cc:377] Logged in from keytab as kudu/127.2.2.3@KRBTEST.COM (short username kudu)
I20260619 14:35:46.411212 18864 webserver.cc:460] Webserver started at http://127.2.2.3:44157/ using document root <none> and password file <none>
I20260619 14:35:46.411770 18864 fs_manager.cc:359] Metadata directory not provided
I20260619 14:35:46.411834 18864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:35:46.415572 18864 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:35:46.417572 18879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:35:46.418817 18864 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:35:46.418949 18864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-2/wal
uuid: "8d1b5a5b42be462895944884b9163686"
format_stamp: "Formatted at 2026-06-19 14:35:04 on dist-test-slave-70rm"
server_key: "01fabbf4a4bacd1958c69260620e5990"
server_key_iv: "255e76cd5e1a3ae84d6154042addcaa4"
server_key_version: "encryptionkey@0"
I20260619 14:35:46.419486 18864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:35:46.437698 18864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:35:46.440763 18864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:35:46.440963 18864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:35:46.441557 18864 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:35:46.442458 18864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:35:46.442523 18864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:35:46.442595 18864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:35:46.442641 18864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:35:46.451566 18864 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.3:45385
I20260619 14:35:46.451588 18992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.3:45385 every 8 connection(s)
I20260619 14:35:46.452586 18864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
I20260619 14:35:46.456439 18995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:46.454757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41891 (local address 127.2.2.3:41097)
0619 14:35:46.455199 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:46.455309 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:41891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":113,"threads_started":1}
I20260619 14:35:46.456684 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 18864
W20260619 14:35:46.456912 18993 heartbeater.cc:644] Failed to heartbeat to 127.2.2.62:41891 (0 consecutive failures): Network error: Failed to ping master at 127.2.2.62:41891: Client connection negotiation failed: client connection to 127.2.2.62:41891: connect: Connection refused (error 111)
I20260619 14:35:46.457479 18995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:46.457263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41891 (local address 127.2.2.3:33971)
0619 14:35:46.457373 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:46.457428 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:41891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 14:35:46.458038 18995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:46.457862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41891 (local address 127.2.2.3:51281)
0619 14:35:46.457955 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:46.457994 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:41891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 14:35:46.458271 18993 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 14:35:47.095654 18997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:47.095190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41891 (local address 127.2.2.1:35391)
0619 14:35:47.095490 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:47.095545 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:41891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":107,"threads_started":1}
I20260619 14:35:47.242719 18998 raft_consensus.cc:482] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:35:47.242938 18998 raft_consensus.cc:504] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ea989b097c44838b4491edae66ac67" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 33101 } } peers { permanent_uuid: "8d1b5a5b42be462895944884b9163686" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45385 } } peers { permanent_uuid: "ed48fbe415dd43babd6582e39e83e02c" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 36581 } }
I20260619 14:35:47.244097 18998 leader_election.cc:288] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37ea989b097c44838b4491edae66ac67 (127.2.2.2:33101), 8d1b5a5b42be462895944884b9163686 (127.2.2.3:45385)
Jun 19 14:35:47 dist-test-slave-70rm krb5kdc[17415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879745, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.1@KRBTEST.COM for kudu/127.2.2.3@KRBTEST.COM
Jun 19 14:35:47 dist-test-slave-70rm krb5kdc[17415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879745, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.1@KRBTEST.COM for kudu/127.2.2.2@KRBTEST.COM
I20260619 14:35:47.262054 19001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:47.244538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:41095 (local address 127.2.2.3:45385)
0619 14:35:47.244961 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0619 14:35:47.244980 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 14:35:47.245030 (+ 50us) server_negotiation.cc:407] Connection header received
0619 14:35:47.245893 (+ 863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:35:47.245901 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:35:47.246408 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:35:47.246750 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:35:47.248720 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:35:47.250387 (+ 1667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:35:47.252377 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:35:47.253145 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:35:47.253327 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:35:47.257024 (+ 3697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:35:47.257032 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:35:47.257050 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:35:47.257087 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:35:47.259292 (+ 2205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:35:47.260023 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:35:47.260036 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:35:47.260041 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:35:47.260149 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:35:47.260513 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:35:47.260518 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:35:47.260520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:35:47.260942 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:35:47.261076 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 14:35:47.261601 (+ 525us) server_negotiation.cc:299] Negotiation successful
0619 14:35:47.261859 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":109,"threads_started":1}
I20260619 14:35:47.262071 18999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:47.244366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.3:45385 (local address 127.2.2.1:41095)
0619 14:35:47.244719 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:47.244733 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:35:47.244854 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:35:47.246865 (+ 2011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:35:47.246871 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:35:47.247538 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:35:47.248489 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:35:47.248511 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:35:47.250566 (+ 2055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:35:47.250570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:35:47.252221 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:35:47.252234 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:35:47.253277 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:35:47.253303 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:35:47.253599 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:35:47.254431 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:35:47.254459 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:35:47.256748 (+ 2289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:35:47.259392 (+ 2644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:35:47.259395 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:35:47.259396 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:35:47.259872 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:35:47.260256 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:35:47.260259 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:35:47.260261 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:35:47.260416 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:35:47.261067 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:35:47.261073 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:35:47.261286 (+ 213us) client_negotiation.cc:769] Sending connection context
0619 14:35:47.261480 (+ 194us) client_negotiation.cc:240] Negotiation successful
0619 14:35:47.261837 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":285,"spinlock_wait_cycles":16640,"thread_start_us":75,"threads_started":1}
I20260619 14:35:47.262091 19000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:47.244506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:39457 (local address 127.2.2.2:33101)
0619 14:35:47.245101 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0619 14:35:47.245112 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:35:47.245144 (+ 32us) server_negotiation.cc:407] Connection header received
0619 14:35:47.245893 (+ 749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:35:47.245901 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:35:47.246408 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:35:47.246750 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:35:47.248720 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:35:47.250387 (+ 1667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:35:47.252377 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:35:47.253146 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:35:47.253327 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:35:47.257188 (+ 3861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:35:47.257193 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:35:47.257205 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:35:47.257234 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:35:47.259219 (+ 1985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:35:47.259993 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:35:47.259999 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:35:47.260005 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:35:47.260150 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:35:47.260529 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:35:47.260532 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:35:47.260534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:35:47.261006 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:35:47.261147 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 14:35:47.261594 (+ 447us) server_negotiation.cc:299] Negotiation successful
0619 14:35:47.261893 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":88,"threads_started":1}
I20260619 14:35:47.262149 18997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:47.244302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.2:33101 (local address 127.2.2.1:39457)
0619 14:35:47.244481 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:47.244496 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:35:47.244608 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:35:47.246865 (+ 2257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:35:47.246871 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:35:47.247538 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:35:47.248489 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:35:47.248511 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:35:47.250568 (+ 2057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:35:47.250571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:35:47.252221 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:35:47.252234 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:35:47.253277 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:35:47.253303 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:35:47.253586 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:35:47.254431 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:35:47.254454 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:35:47.257001 (+ 2547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:35:47.259360 (+ 2359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:35:47.259369 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:35:47.259385 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:35:47.259872 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:35:47.260269 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:35:47.260272 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:35:47.260274 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:35:47.260425 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:35:47.261133 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:35:47.261136 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:35:47.261290 (+ 154us) client_negotiation.cc:769] Sending connection context
0619 14:35:47.261480 (+ 190us) client_negotiation.cc:240] Negotiation successful
0619 14:35:47.261926 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":638,"spinlock_wait_cycles":22400}
W20260619 14:35:47.264730 18616 leader_election.cc:341] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8d1b5a5b42be462895944884b9163686 (127.2.2.3:45385): Not found: Tablet not found: aadacf5a44c743d280964d0ae1e52f23
W20260619 14:35:47.264935 18614 leader_election.cc:341] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 37ea989b097c44838b4491edae66ac67 (127.2.2.2:33101): Not found: Tablet not found: aadacf5a44c743d280964d0ae1e52f23
I20260619 14:35:47.265034 18614 leader_election.cc:302] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ed48fbe415dd43babd6582e39e83e02c; no voters: 37ea989b097c44838b4491edae66ac67, 8d1b5a5b42be462895944884b9163686
I20260619 14:35:47.265314 18998 raft_consensus.cc:2726] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 14:35:47.265736 19002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:47.265330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41891 (local address 127.2.2.2:44093)
0619 14:35:47.265605 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:47.265653 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:41891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":67,"threads_started":1}
I20260619 14:35:47.459159 19003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:47.458788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41891 (local address 127.2.2.3:34981)
0619 14:35:47.459011 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:47.459069 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:41891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":68,"threads_started":1}
I20260619 14:35:48.096939 19004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:48.096478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41891 (local address 127.2.2.1:41211)
0619 14:35:48.096771 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:48.096842 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:41891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":102,"threads_started":1}
I20260619 14:35:48.266973 19005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:48.266552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41891 (local address 127.2.2.2:42079)
0619 14:35:48.266806 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:48.266866 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:41891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":73,"threads_started":1}
I20260619 14:35:48.460389 19006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:48.459892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41891 (local address 127.2.2.3:59003)
0619 14:35:48.460154 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:48.460216 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:41891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":73,"threads_started":1}
I20260619 14:35:49.098318 19007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:49.097778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41891 (local address 127.2.2.1:45123)
0619 14:35:49.098087 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:49.098194 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:41891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":116,"threads_started":1}
I20260619 14:35:49.268136 19008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:49.267712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41891 (local address 127.2.2.2:43173)
0619 14:35:49.267969 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:49.268026 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:41891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":90,"threads_started":1}
I20260619 14:35:49.361481 19009 raft_consensus.cc:482] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:35:49.361639 19009 raft_consensus.cc:504] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ea989b097c44838b4491edae66ac67" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 33101 } } peers { permanent_uuid: "8d1b5a5b42be462895944884b9163686" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45385 } } peers { permanent_uuid: "ed48fbe415dd43babd6582e39e83e02c" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 36581 } }
I20260619 14:35:49.362176 19009 leader_election.cc:288] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37ea989b097c44838b4491edae66ac67 (127.2.2.2:33101), 8d1b5a5b42be462895944884b9163686 (127.2.2.3:45385)
W20260619 14:35:49.363041 18616 leader_election.cc:341] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8d1b5a5b42be462895944884b9163686 (127.2.2.3:45385): Not found: Tablet not found: aadacf5a44c743d280964d0ae1e52f23
W20260619 14:35:49.363150 18614 leader_election.cc:341] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 37ea989b097c44838b4491edae66ac67 (127.2.2.2:33101): Not found: Tablet not found: aadacf5a44c743d280964d0ae1e52f23
I20260619 14:35:49.363277 18614 leader_election.cc:302] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ed48fbe415dd43babd6582e39e83e02c; no voters: 37ea989b097c44838b4491edae66ac67, 8d1b5a5b42be462895944884b9163686
I20260619 14:35:49.363456 19009 raft_consensus.cc:2726] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 14:35:49.461520 19010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:49.461145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41891 (local address 127.2.2.3:33951)
0619 14:35:49.461375 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:49.461429 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:41891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":71,"threads_started":1}
I20260619 14:35:50.099489 19011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:50.099070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41891 (local address 127.2.2.1:46839)
0619 14:35:50.099330 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:50.099393 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:41891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":75,"threads_started":1}
I20260619 14:35:50.269332 19012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:50.268926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41891 (local address 127.2.2.2:39573)
0619 14:35:50.269167 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:50.269218 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:41891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":71,"threads_started":1}
I20260619 14:35:50.462765 19013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:50.462323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41891 (local address 127.2.2.3:46317)
0619 14:35:50.462597 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:50.462665 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:41891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":77,"threads_started":1}
I20260619 14:35:51.100790 19014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:51.100359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41891 (local address 127.2.2.1:55647)
0619 14:35:51.100637 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:51.100692 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:41891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":86,"threads_started":1}
I20260619 14:35:51.270699 19015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:51.270274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41891 (local address 127.2.2.2:55305)
0619 14:35:51.270542 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:51.270602 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:41891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":85,"threads_started":1}
I20260619 14:35:51.464063 19016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:51.463611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41891 (local address 127.2.2.3:52055)
0619 14:35:51.463903 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:51.463965 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:41891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":88,"threads_started":1}
I20260619 14:35:51.951913 19017 raft_consensus.cc:482] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:35:51.952069 19017 raft_consensus.cc:504] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ea989b097c44838b4491edae66ac67" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 33101 } } peers { permanent_uuid: "8d1b5a5b42be462895944884b9163686" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45385 } } peers { permanent_uuid: "ed48fbe415dd43babd6582e39e83e02c" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 36581 } }
I20260619 14:35:51.952572 19017 leader_election.cc:288] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37ea989b097c44838b4491edae66ac67 (127.2.2.2:33101), 8d1b5a5b42be462895944884b9163686 (127.2.2.3:45385)
W20260619 14:35:51.953347 18614 leader_election.cc:341] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 37ea989b097c44838b4491edae66ac67 (127.2.2.2:33101): Not found: Tablet not found: aadacf5a44c743d280964d0ae1e52f23
W20260619 14:35:51.953481 18616 leader_election.cc:341] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8d1b5a5b42be462895944884b9163686 (127.2.2.3:45385): Not found: Tablet not found: aadacf5a44c743d280964d0ae1e52f23
I20260619 14:35:51.953613 18616 leader_election.cc:302] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ed48fbe415dd43babd6582e39e83e02c; no voters: 37ea989b097c44838b4491edae66ac67, 8d1b5a5b42be462895944884b9163686
I20260619 14:35:51.953789 19017 raft_consensus.cc:2726] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 14:35:52.102036 19018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:52.101634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41891 (local address 127.2.2.1:49933)
0619 14:35:52.101881 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:52.101939 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:41891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":73,"threads_started":1}
I20260619 14:35:52.272004 19019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:52.271541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41891 (local address 127.2.2.2:51137)
0619 14:35:52.271799 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:52.271890 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:41891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":86,"threads_started":1}
I20260619 14:35:52.465262 19020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:52.464739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41891 (local address 127.2.2.3:58107)
0619 14:35:52.465039 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:52.465147 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:41891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":95,"threads_started":1}
I20260619 14:35:53.103240 19021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:53.102815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41891 (local address 127.2.2.1:52587)
0619 14:35:53.103088 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:53.103145 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:41891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":86,"threads_started":1}
I20260619 14:35:53.273262 19022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:53.272801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41891 (local address 127.2.2.2:36853)
0619 14:35:53.273100 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:53.273165 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:41891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":77,"threads_started":1}
I20260619 14:35:53.466760 19023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:53.466291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41891 (local address 127.2.2.3:58409)
0619 14:35:53.466599 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:53.466664 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:41891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":76,"threads_started":1}
I20260619 14:35:54.104374 19024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:54.103967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41891 (local address 127.2.2.1:57749)
0619 14:35:54.104214 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:54.104279 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:41891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":74,"threads_started":1}
I20260619 14:35:54.274379 19025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:54.273942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41891 (local address 127.2.2.2:46067)
0619 14:35:54.274225 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:54.274284 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:41891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":71,"threads_started":1}
I20260619 14:35:54.468200 19026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:54.467748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41891 (local address 127.2.2.3:34059)
0619 14:35:54.468031 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:54.468097 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:41891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":87,"threads_started":1}
I20260619 14:35:55.105747 19027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:55.105295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41891 (local address 127.2.2.1:52619)
0619 14:35:55.105558 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:55.105641 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:41891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":99,"threads_started":1}
I20260619 14:35:55.275544 19028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:55.275115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41891 (local address 127.2.2.2:46665)
0619 14:35:55.275379 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:55.275439 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:41891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":75,"threads_started":1}
I20260619 14:35:55.469436 19029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:55.469034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41891 (local address 127.2.2.3:48875)
0619 14:35:55.469288 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:55.469343 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:41891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":75,"threads_started":1}
I20260619 14:35:56.107064 19030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:56.106672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41891 (local address 127.2.2.1:37287)
0619 14:35:56.106915 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:56.106971 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:41891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":80,"threads_started":1}
I20260619 14:35:56.276840 19031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:56.276417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41891 (local address 127.2.2.2:59755)
0619 14:35:56.276680 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:56.276740 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:41891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":97,"threads_started":1}
I20260619 14:35:56.470835 19032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:56.470375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41891 (local address 127.2.2.3:39459)
0619 14:35:56.470664 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:56.470733 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:41891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":73,"threads_started":1}
I20260619 14:35:56.720336 19033 raft_consensus.cc:482] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:35:56.720517 19033 raft_consensus.cc:504] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ea989b097c44838b4491edae66ac67" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 33101 } } peers { permanent_uuid: "8d1b5a5b42be462895944884b9163686" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45385 } } peers { permanent_uuid: "ed48fbe415dd43babd6582e39e83e02c" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 36581 } }
I20260619 14:35:56.721032 19033 leader_election.cc:288] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37ea989b097c44838b4491edae66ac67 (127.2.2.2:33101), 8d1b5a5b42be462895944884b9163686 (127.2.2.3:45385)
W20260619 14:35:56.721832 18614 leader_election.cc:341] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 37ea989b097c44838b4491edae66ac67 (127.2.2.2:33101): Not found: Tablet not found: aadacf5a44c743d280964d0ae1e52f23
W20260619 14:35:56.721942 18616 leader_election.cc:341] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8d1b5a5b42be462895944884b9163686 (127.2.2.3:45385): Not found: Tablet not found: aadacf5a44c743d280964d0ae1e52f23
I20260619 14:35:56.722060 18616 leader_election.cc:302] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ed48fbe415dd43babd6582e39e83e02c; no voters: 37ea989b097c44838b4491edae66ac67, 8d1b5a5b42be462895944884b9163686
I20260619 14:35:56.722303 19033 raft_consensus.cc:2726] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 14:35:57.108438 19034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:57.107976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41891 (local address 127.2.2.1:52519)
0619 14:35:57.108277 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:57.108340 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:41891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":111,"threads_started":1}
I20260619 14:35:57.278326 19035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:57.277785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41891 (local address 127.2.2.2:55263)
0619 14:35:57.278092 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:57.278222 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:41891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":90,"threads_started":1}
I20260619 14:35:57.472146 19036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:57.471661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41891 (local address 127.2.2.3:39111)
0619 14:35:57.471923 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:57.472000 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:41891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":85,"threads_started":1}
I20260619 14:35:58.109721 19037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:58.109296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41891 (local address 127.2.2.1:59529)
0619 14:35:58.109562 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:58.109625 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:41891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":75,"threads_started":1}
I20260619 14:35:58.279560 19038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:58.279151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41891 (local address 127.2.2.2:36527)
0619 14:35:58.279406 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:58.279464 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:41891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":76,"threads_started":1}
I20260619 14:35:58.385633 19039 raft_consensus.cc:482] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:35:58.385816 19039 raft_consensus.cc:504] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ea989b097c44838b4491edae66ac67" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 33101 } } peers { permanent_uuid: "8d1b5a5b42be462895944884b9163686" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45385 } } peers { permanent_uuid: "ed48fbe415dd43babd6582e39e83e02c" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 36581 } }
I20260619 14:35:58.386337 19039 leader_election.cc:288] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37ea989b097c44838b4491edae66ac67 (127.2.2.2:33101), 8d1b5a5b42be462895944884b9163686 (127.2.2.3:45385)
W20260619 14:35:58.387164 18614 leader_election.cc:341] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 37ea989b097c44838b4491edae66ac67 (127.2.2.2:33101): Not found: Tablet not found: aadacf5a44c743d280964d0ae1e52f23
W20260619 14:35:58.387282 18616 leader_election.cc:341] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8d1b5a5b42be462895944884b9163686 (127.2.2.3:45385): Not found: Tablet not found: aadacf5a44c743d280964d0ae1e52f23
I20260619 14:35:58.387395 18616 leader_election.cc:302] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ed48fbe415dd43babd6582e39e83e02c; no voters: 37ea989b097c44838b4491edae66ac67, 8d1b5a5b42be462895944884b9163686
I20260619 14:35:58.387579 19039 raft_consensus.cc:2726] T aadacf5a44c743d280964d0ae1e52f23 P ed48fbe415dd43babd6582e39e83e02c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 14:35:58.473435 19040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:58.473000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41891 (local address 127.2.2.3:51433)
0619 14:35:58.473278 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:58.473339 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:41891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":93,"threads_started":1}
I20260619 14:35:59.111058 19041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:59.110641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41891 (local address 127.2.2.1:54401)
0619 14:35:59.110899 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:59.110959 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:41891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":78,"threads_started":1}
I20260619 14:35:59.280722 19042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:59.280325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41891 (local address 127.2.2.2:33827)
0619 14:35:59.280572 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:59.280628 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:41891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":70,"threads_started":1}
I20260619 14:35:59.474647 19043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:35:59.474270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41891 (local address 127.2.2.3:45871)
0619 14:35:59.474497 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0619 14:35:59.474552 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:41891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":68,"threads_started":1}
I20260619 14:36:00.112300 19044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:00.111865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41891 (local address 127.2.2.1:37439)
0619 14:36:00.112140 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:00.112196 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:41891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":86,"threads_started":1}
I20260619 14:36:00.282044 19045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:00.281608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41891 (local address 127.2.2.2:40085)
0619 14:36:00.281880 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:00.281944 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:41891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":80,"threads_started":1}
I20260619 14:36:00.475841 19046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:00.475422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41891 (local address 127.2.2.3:51507)
0619 14:36:00.475660 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:00.475716 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:41891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":76,"threads_started":1}
I20260619 14:36:01.113716 19047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:01.113307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41891 (local address 127.2.2.1:53151)
0619 14:36:01.113566 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:01.113622 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:41891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":74,"threads_started":1}
I20260619 14:36:01.283407 19048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:01.282985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41891 (local address 127.2.2.2:47537)
0619 14:36:01.283231 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:01.283310 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.62:41891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":77,"threads_started":1}
I20260619 14:36:01.462240 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 18564
I20260619 14:36:01.468081 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 18730
I20260619 14:36:01.473762 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 18864
2026-06-19T14:36:01Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59418 ms)
[----------] 30 tests from SecurityITest (246474 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:36:01 dist-test-slave-70rm krb5kdc[19051](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:36:01 dist-test-slave-70rm krb5kdc[19051](info): set up 2 sockets
Jun 19 14:36:01 dist-test-slave-70rm krb5kdc[19051](info): commencing operation
krb5kdc: starting...
W20260619 14:36:02.522033 2056 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Jun 19 14:36:02 dist-test-slave-70rm krb5kdc[19051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:36:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:36:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.2.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab.
Entry for principal kudu/127.2.2.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab.
I20260619 14:36:02.646107 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:36969
--webserver_interface=127.2.2.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:46583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.2.62:36969
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:36:02.756989 19064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:36:02.757339 19064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:36:02.757422 19064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:36:02.761622 19064 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:36:02.761731 19064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:36:02.761780 19064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:36:02.761843 19064 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:36:02.761885 19064 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:36:02.767491 19064 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:46583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.2.62:36969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:36969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.2.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:36:02.768908 19064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:36:02.770093 19064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:36:02.779142 19069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:02.779287 19064 server_base.cc:1034] running on GCE node
W20260619 14:36:02.779217 19070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:02.779142 19072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:02.779986 19064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:36:02.781142 19064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:36:02.782320 19064 hybrid_clock.cc:648] HybridClock initialized: now 1781879762782288 us; error 44 us; skew 500 ppm
Jun 19 14:36:02 dist-test-slave-70rm krb5kdc[19051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879762, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:36:02.785123 19064 init.cc:377] Logged in from keytab as kudu/127.2.2.62@KRBTEST.COM (short username kudu)
I20260619 14:36:02.786017 19064 webserver.cc:460] Webserver started at http://127.2.2.62:35721/ using document root <none> and password file <none>
I20260619 14:36:02.786651 19064 fs_manager.cc:359] Metadata directory not provided
I20260619 14:36:02.786720 19064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:36:02.786887 19064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:36:02.788924 19064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/master-0/data/instance:
uuid: "418c6192aaba424b8b3abb4c29e0e98c"
format_stamp: "Formatted at 2026-06-19 14:36:02 on dist-test-slave-70rm"
server_key: "bb77408232416307b0f53a2e7b7e710f"
server_key_iv: "e458efdafb276b07bd1d92c9551c9335"
server_key_version: "encryptionkey@0"
I20260619 14:36:02.789431 19064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/master-0/wal/instance:
uuid: "418c6192aaba424b8b3abb4c29e0e98c"
format_stamp: "Formatted at 2026-06-19 14:36:02 on dist-test-slave-70rm"
server_key: "bb77408232416307b0f53a2e7b7e710f"
server_key_iv: "e458efdafb276b07bd1d92c9551c9335"
server_key_version: "encryptionkey@0"
I20260619 14:36:02.793078 19064 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 14:36:02.795553 19079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:02.796839 19064 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:36:02.796947 19064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/master-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/master-0/wal
uuid: "418c6192aaba424b8b3abb4c29e0e98c"
format_stamp: "Formatted at 2026-06-19 14:36:02 on dist-test-slave-70rm"
server_key: "bb77408232416307b0f53a2e7b7e710f"
server_key_iv: "e458efdafb276b07bd1d92c9551c9335"
server_key_version: "encryptionkey@0"
I20260619 14:36:02.797034 19064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:36:02.808514 19064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:36:02.811350 19064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:36:02.811501 19064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:36:02.818408 19064 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.62:36969
I20260619 14:36:02.818434 19131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.62:36969 every 8 connection(s)
I20260619 14:36:02.819365 19064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
I20260619 14:36:02.822026 19132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:36:02.822429 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 19064
I20260619 14:36:02.822553 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/master-0/wal/instance
I20260619 14:36:02.822852 2056 external_mini_cluster.cc:1428] Setting key 915d6aa8186b492d9adf100451545b25
I20260619 14:36:02.829012 19132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 418c6192aaba424b8b3abb4c29e0e98c: Bootstrap starting.
I20260619 14:36:02.831853 19132 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 418c6192aaba424b8b3abb4c29e0e98c: Neither blocks nor log segments found. Creating new log.
Jun 19 14:36:02 dist-test-slave-70rm krb5kdc[19051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:36:02.832593 19132 log.cc:824] T 00000000000000000000000000000000 P 418c6192aaba424b8b3abb4c29e0e98c: Log is configured to *not* fsync() on all Append() calls
I20260619 14:36:02.835022 19132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 418c6192aaba424b8b3abb4c29e0e98c: No bootstrap required, opened a new log
I20260619 14:36:02.837735 19135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:02.824189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36670 (local address 127.2.2.62:36969)
0619 14:36:02.824605 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:02.824622 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:02.824657 (+ 35us) server_negotiation.cc:407] Connection header received
0619 14:36:02.825329 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:02.825345 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:02.825695 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:02.825974 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:02.826933 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:02.827926 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:02.829090 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:02.829610 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:02.829729 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:02.832533 (+ 2804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:02.832539 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:02.832553 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:36:02.832583 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:02.834601 (+ 2018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:02.835036 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:02.835044 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:02.835051 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:02.835191 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:02.835462 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:02.835465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:02.835467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:02.835961 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:02.836153 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:02.836371 (+ 218us) server_negotiation.cc:299] Negotiation successful
0619 14:36:02.836671 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":136,"threads_started":1}
I20260619 14:36:02.839530 19132 raft_consensus.cc:348] T 00000000000000000000000000000000 P 418c6192aaba424b8b3abb4c29e0e98c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "418c6192aaba424b8b3abb4c29e0e98c" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 36969 } }
I20260619 14:36:02.839782 19132 raft_consensus.cc:374] T 00000000000000000000000000000000 P 418c6192aaba424b8b3abb4c29e0e98c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:36:02.839857 19132 raft_consensus.cc:729] T 00000000000000000000000000000000 P 418c6192aaba424b8b3abb4c29e0e98c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 418c6192aaba424b8b3abb4c29e0e98c, State: Initialized, Role: FOLLOWER
I20260619 14:36:02.840317 19132 consensus_queue.cc:260] T 00000000000000000000000000000000 P 418c6192aaba424b8b3abb4c29e0e98c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "418c6192aaba424b8b3abb4c29e0e98c" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 36969 } }
I20260619 14:36:02.840440 19132 raft_consensus.cc:388] T 00000000000000000000000000000000 P 418c6192aaba424b8b3abb4c29e0e98c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:36:02.840529 19132 raft_consensus.cc:482] T 00000000000000000000000000000000 P 418c6192aaba424b8b3abb4c29e0e98c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:36:02.840646 19132 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 418c6192aaba424b8b3abb4c29e0e98c [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:36:02.841571 19132 raft_consensus.cc:504] T 00000000000000000000000000000000 P 418c6192aaba424b8b3abb4c29e0e98c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "418c6192aaba424b8b3abb4c29e0e98c" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 36969 } }
I20260619 14:36:02.841915 19132 leader_election.cc:302] T 00000000000000000000000000000000 P 418c6192aaba424b8b3abb4c29e0e98c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 418c6192aaba424b8b3abb4c29e0e98c; no voters:
I20260619 14:36:02.842284 19132 leader_election.cc:288] T 00000000000000000000000000000000 P 418c6192aaba424b8b3abb4c29e0e98c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:36:02.842352 19137 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 418c6192aaba424b8b3abb4c29e0e98c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:36:02.842628 19137 raft_consensus.cc:686] T 00000000000000000000000000000000 P 418c6192aaba424b8b3abb4c29e0e98c [term 1 LEADER]: Becoming Leader. State: Replica: 418c6192aaba424b8b3abb4c29e0e98c, State: Running, Role: LEADER
I20260619 14:36:02.842985 19137 consensus_queue.cc:237] T 00000000000000000000000000000000 P 418c6192aaba424b8b3abb4c29e0e98c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "418c6192aaba424b8b3abb4c29e0e98c" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 36969 } }
I20260619 14:36:02.843528 19132 sys_catalog.cc:564] T 00000000000000000000000000000000 P 418c6192aaba424b8b3abb4c29e0e98c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:36:02.844569 19138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 418c6192aaba424b8b3abb4c29e0e98c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "418c6192aaba424b8b3abb4c29e0e98c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "418c6192aaba424b8b3abb4c29e0e98c" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 36969 } } }
I20260619 14:36:02.844688 19138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 418c6192aaba424b8b3abb4c29e0e98c [sys.catalog]: This master's current role is: LEADER
I20260619 14:36:02.845068 19146 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:36:02.845074 19139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 418c6192aaba424b8b3abb4c29e0e98c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 418c6192aaba424b8b3abb4c29e0e98c. Latest consensus state: current_term: 1 leader_uuid: "418c6192aaba424b8b3abb4c29e0e98c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "418c6192aaba424b8b3abb4c29e0e98c" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 36969 } } }
I20260619 14:36:02.845778 19139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 418c6192aaba424b8b3abb4c29e0e98c [sys.catalog]: This master's current role is: LEADER
I20260619 14:36:02.848177 19146 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:36:02.853977 19146 catalog_manager.cc:1347] Generated new cluster ID: 1aa64bd3beb84f3eb32da2d2fd77bfdd
I20260619 14:36:02.854046 19146 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:36:02.872843 19146 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:36:02.874234 19146 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:36:02.880947 19146 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 418c6192aaba424b8b3abb4c29e0e98c: Generated new TSK 0
I20260619 14:36:02.881825 19146 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.2.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab.
Entry for principal kudu/127.2.2.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab.
I20260619 14:36:02.911221 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.1:0
--local_ip_for_outbound_sockets=127.2.2.1
--webserver_interface=127.2.2.1
--webserver_port=0
--tserver_master_addrs=127.2.2.62:36969
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab
--principal=kudu/127.2.2.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:46583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:36:03.022513 19158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:36:03.022792 19158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:36:03.022876 19158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:36:03.027017 19158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:36:03.027074 19158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:36:03.027179 19158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.1
I20260619 14:36:03.032473 19158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:46583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab
--principal=kudu/127.2.2.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.2.1
--webserver_port=0
--tserver_master_addrs=127.2.2.62:36969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.1
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:36:03.033555 19158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:36:03.034683 19158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:36:03.044538 19163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:03.044503 19164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:03.044515 19166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:03.044777 19158 server_base.cc:1034] running on GCE node
I20260619 14:36:03.045362 19158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:36:03.046021 19158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:36:03.047206 19158 hybrid_clock.cc:648] HybridClock initialized: now 1781879763047196 us; error 35 us; skew 500 ppm
Jun 19 14:36:03 dist-test-slave-70rm krb5kdc[19051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879763, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:36:03.049885 19158 init.cc:377] Logged in from keytab as kudu/127.2.2.1@KRBTEST.COM (short username kudu)
I20260619 14:36:03.050793 19158 webserver.cc:460] Webserver started at http://127.2.2.1:42985/ using document root <none> and password file <none>
I20260619 14:36:03.051362 19158 fs_manager.cc:359] Metadata directory not provided
I20260619 14:36:03.051455 19158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:36:03.051689 19158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:36:03.053704 19158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/ts-0/data/instance:
uuid: "d2dec4eb13714c4b8e912118d35961b9"
format_stamp: "Formatted at 2026-06-19 14:36:03 on dist-test-slave-70rm"
server_key: "e9c46159c602eb86697f7114f7e6294b"
server_key_iv: "cb4171a8ffae9178ab6efff29caac147"
server_key_version: "encryptionkey@0"
I20260619 14:36:03.054335 19158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/ts-0/wal/instance:
uuid: "d2dec4eb13714c4b8e912118d35961b9"
format_stamp: "Formatted at 2026-06-19 14:36:03 on dist-test-slave-70rm"
server_key: "e9c46159c602eb86697f7114f7e6294b"
server_key_iv: "cb4171a8ffae9178ab6efff29caac147"
server_key_version: "encryptionkey@0"
I20260619 14:36:03.057991 19158 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 14:36:03.060447 19173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:03.061614 19158 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:36:03.061744 19158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/ts-0/wal
uuid: "d2dec4eb13714c4b8e912118d35961b9"
format_stamp: "Formatted at 2026-06-19 14:36:03 on dist-test-slave-70rm"
server_key: "e9c46159c602eb86697f7114f7e6294b"
server_key_iv: "cb4171a8ffae9178ab6efff29caac147"
server_key_version: "encryptionkey@0"
I20260619 14:36:03.061858 19158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:36:03.087196 19158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:36:03.090317 19158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:36:03.090467 19158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:36:03.090991 19158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:36:03.091768 19158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:36:03.091814 19158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:03.091854 19158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:36:03.091874 19158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:03.101374 19158 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.1:43841
I20260619 14:36:03.101454 19286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.1:43841 every 8 connection(s)
I20260619 14:36:03.102435 19158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
I20260619 14:36:03.107986 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 19158
I20260619 14:36:03.108130 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/ts-0/wal/instance
I20260619 14:36:03.108412 2056 external_mini_cluster.cc:1428] Setting key c3ee4b73ec28c1ac43555b3eddcc0361
Jun 19 14:36:03 dist-test-slave-70rm krb5kdc[19051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879763, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.1@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:36:03.117069 19135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:03.104287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:44453 (local address 127.2.2.62:36969)
0619 14:36:03.104425 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:03.104435 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:03.105287 (+ 852us) server_negotiation.cc:407] Connection header received
0619 14:36:03.106257 (+ 970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:03.106261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:03.106324 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:03.106423 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:03.107620 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:03.108258 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:03.109485 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:03.109728 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:03.109840 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:03.112478 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:03.112485 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:03.112491 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:36:03.112526 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:03.114507 (+ 1981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:03.115110 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:03.115116 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:03.115121 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:03.115234 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:03.115649 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:03.115656 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:03.115659 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:03.115968 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:03.116084 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:03.116684 (+ 600us) server_negotiation.cc:299] Negotiation successful
0619 14:36:03.116853 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 14:36:03.117904 19289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:03.104677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:36969 (local address 127.2.2.1:44453)
0619 14:36:03.105131 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:03.105174 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 14:36:03.106002 (+ 828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:36:03.106605 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:36:03.106613 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:36:03.106993 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:36:03.107470 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:03.107483 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:03.108381 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:03.108385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:03.109356 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:03.109366 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:03.109827 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:03.109884 (+ 57us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:03.110057 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:03.110725 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:36:03.110742 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:36:03.112292 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:36:03.114664 (+ 2372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:36:03.114670 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:36:03.114685 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:36:03.114988 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:36:03.115361 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:36:03.115364 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:36:03.115366 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:36:03.115543 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:36:03.116138 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:36:03.116143 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:36:03.116413 (+ 270us) client_negotiation.cc:769] Sending connection context
0619 14:36:03.116592 (+ 179us) client_negotiation.cc:240] Negotiation successful
0619 14:36:03.116890 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":112,"threads_started":1}
WARNING: no policy specified for kudu/127.2.2.2@KRBTEST.COM; defaulting to no policy
I20260619 14:36:03.119467 19287 heartbeater.cc:347] Connected to a master server at 127.2.2.62:36969
I20260619 14:36:03.119761 19287 heartbeater.cc:464] Registering TS with master...
I20260619 14:36:03.120319 19287 heartbeater.cc:511] Master 127.2.2.62:36969 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.2@KRBTEST.COM" created.
I20260619 14:36:03.122001 19096 ts_manager.cc:194] Registered new tserver with Master: d2dec4eb13714c4b8e912118d35961b9 (127.2.2.1:43841)
I20260619 14:36:03.123297 19096 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.2.1@KRBTEST.COM'} at 127.2.2.1:44453
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab.
Entry for principal kudu/127.2.2.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab.
I20260619 14:36:03.136686 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.2:0
--local_ip_for_outbound_sockets=127.2.2.2
--webserver_interface=127.2.2.2
--webserver_port=0
--tserver_master_addrs=127.2.2.62:36969
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab
--principal=kudu/127.2.2.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:46583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:36:03.247186 19292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:36:03.247467 19292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:36:03.247550 19292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:36:03.252049 19292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:36:03.252156 19292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:36:03.252259 19292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.2
I20260619 14:36:03.257773 19292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:46583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab
--principal=kudu/127.2.2.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.2.2
--webserver_port=0
--tserver_master_addrs=127.2.2.62:36969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.2
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:36:03.258993 19292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:36:03.260063 19292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:36:03.270228 19300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:03.270306 19292 server_base.cc:1034] running on GCE node
W20260619 14:36:03.270228 19297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:03.270232 19298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:03.271059 19292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:36:03.271733 19292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:36:03.272945 19292 hybrid_clock.cc:648] HybridClock initialized: now 1781879763272908 us; error 50 us; skew 500 ppm
Jun 19 14:36:03 dist-test-slave-70rm krb5kdc[19051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879763, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:36:03.276098 19292 init.cc:377] Logged in from keytab as kudu/127.2.2.2@KRBTEST.COM (short username kudu)
I20260619 14:36:03.277077 19292 webserver.cc:460] Webserver started at http://127.2.2.2:37173/ using document root <none> and password file <none>
I20260619 14:36:03.277678 19292 fs_manager.cc:359] Metadata directory not provided
I20260619 14:36:03.277791 19292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:36:03.278035 19292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:36:03.280185 19292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/ts-1/data/instance:
uuid: "bd694eb40e0e4910975c82a2496fca2d"
format_stamp: "Formatted at 2026-06-19 14:36:03 on dist-test-slave-70rm"
server_key: "3775d4211ac1cadf21c13d3cdb23c214"
server_key_iv: "e70243a288b82b3427d5258872c865f8"
server_key_version: "encryptionkey@0"
I20260619 14:36:03.280793 19292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/ts-1/wal/instance:
uuid: "bd694eb40e0e4910975c82a2496fca2d"
format_stamp: "Formatted at 2026-06-19 14:36:03 on dist-test-slave-70rm"
server_key: "3775d4211ac1cadf21c13d3cdb23c214"
server_key_iv: "e70243a288b82b3427d5258872c865f8"
server_key_version: "encryptionkey@0"
I20260619 14:36:03.285050 19292 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260619 14:36:03.287710 19307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:03.289090 19292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 14:36:03.289240 19292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/ts-1/wal
uuid: "bd694eb40e0e4910975c82a2496fca2d"
format_stamp: "Formatted at 2026-06-19 14:36:03 on dist-test-slave-70rm"
server_key: "3775d4211ac1cadf21c13d3cdb23c214"
server_key_iv: "e70243a288b82b3427d5258872c865f8"
server_key_version: "encryptionkey@0"
I20260619 14:36:03.289362 19292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:36:03.299268 19292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:36:03.302903 19292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:36:03.303099 19292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:36:03.303692 19292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:36:03.304515 19292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:36:03.304561 19292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:03.304628 19292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:36:03.304670 19292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:03.313580 19292 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.2:38833
I20260619 14:36:03.313601 19420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.2:38833 every 8 connection(s)
I20260619 14:36:03.314534 19292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
I20260619 14:36:03.323071 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 19292
I20260619 14:36:03.323222 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/ts-1/wal/instance
Jun 19 14:36:03 dist-test-slave-70rm krb5kdc[19051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879763, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.2@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:36:03.323553 2056 external_mini_cluster.cc:1428] Setting key 1d5ffe0b30ebe0f50beb1716f109e83e
I20260619 14:36:03.328111 19135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:03.316315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:34491 (local address 127.2.2.62:36969)
0619 14:36:03.316499 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:03.316503 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:03.317368 (+ 865us) server_negotiation.cc:407] Connection header received
0619 14:36:03.318300 (+ 932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:03.318303 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:03.318351 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:03.318435 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:03.319630 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:03.320155 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:03.321258 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:03.321469 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:03.321546 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:03.324154 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:03.324165 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:03.324167 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:36:03.324192 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:03.325671 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:03.326352 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:03.326355 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:03.326356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:03.326437 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:03.326863 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:03.326868 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:03.326870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:03.327139 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:03.327248 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:03.327759 (+ 511us) server_negotiation.cc:299] Negotiation successful
0619 14:36:03.327905 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 14:36:03.329072 19423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:03.316730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:36969 (local address 127.2.2.2:34491)
0619 14:36:03.317175 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:03.317243 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0619 14:36:03.318008 (+ 765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:36:03.318594 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:36:03.318601 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:36:03.318962 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:36:03.319467 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:03.319481 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:03.320269 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:03.320271 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:03.321149 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:03.321157 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:03.321552 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:03.321600 (+ 48us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:03.321737 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:03.322310 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:36:03.322326 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:36:03.323965 (+ 1639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:36:03.325796 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:36:03.325805 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:36:03.325825 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:36:03.326243 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:36:03.326536 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:36:03.326542 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:36:03.326547 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:36:03.326743 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:36:03.327253 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:36:03.327259 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:36:03.327499 (+ 240us) client_negotiation.cc:769] Sending connection context
0619 14:36:03.327651 (+ 152us) client_negotiation.cc:240] Negotiation successful
0619 14:36:03.327913 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":110,"threads_started":1}
I20260619 14:36:03.330377 19421 heartbeater.cc:347] Connected to a master server at 127.2.2.62:36969
I20260619 14:36:03.330649 19421 heartbeater.cc:464] Registering TS with master...
I20260619 14:36:03.331143 19421 heartbeater.cc:511] Master 127.2.2.62:36969 requested a full tablet report, sending...
I20260619 14:36:03.332266 19096 ts_manager.cc:194] Registered new tserver with Master: bd694eb40e0e4910975c82a2496fca2d (127.2.2.2:38833)
I20260619 14:36:03.333120 19096 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.2.2@KRBTEST.COM'} at 127.2.2.2:34491
WARNING: no policy specified for kudu/127.2.2.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab.
Entry for principal kudu/127.2.2.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab.
I20260619 14:36:03.351568 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.3:0
--local_ip_for_outbound_sockets=127.2.2.3
--webserver_interface=127.2.2.3
--webserver_port=0
--tserver_master_addrs=127.2.2.62:36969
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab
--principal=kudu/127.2.2.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:46583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:36:03.462829 19426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:36:03.463147 19426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:36:03.463195 19426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:36:03.467646 19426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:36:03.467718 19426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:36:03.467811 19426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.3
I20260619 14:36:03.473156 19426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:46583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab
--principal=kudu/127.2.2.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.2.3
--webserver_port=0
--tserver_master_addrs=127.2.2.62:36969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.3
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:36:03.474257 19426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:36:03.475286 19426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:36:03.486356 19434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:03.486377 19431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:03.486421 19432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:03.486616 19426 server_base.cc:1034] running on GCE node
I20260619 14:36:03.487046 19426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:36:03.487778 19426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:36:03.488961 19426 hybrid_clock.cc:648] HybridClock initialized: now 1781879763488931 us; error 39 us; skew 500 ppm
Jun 19 14:36:03 dist-test-slave-70rm krb5kdc[19051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879763, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:36:03.491811 19426 init.cc:377] Logged in from keytab as kudu/127.2.2.3@KRBTEST.COM (short username kudu)
I20260619 14:36:03.492688 19426 webserver.cc:460] Webserver started at http://127.2.2.3:44215/ using document root <none> and password file <none>
I20260619 14:36:03.493214 19426 fs_manager.cc:359] Metadata directory not provided
I20260619 14:36:03.493273 19426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:36:03.493582 19426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:36:03.495748 19426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/ts-2/data/instance:
uuid: "3ae43cb481b940bb8be8cc303a2ad1be"
format_stamp: "Formatted at 2026-06-19 14:36:03 on dist-test-slave-70rm"
server_key: "28e3afa99f1dd344004815238aafb607"
server_key_iv: "15aa97968d86b498681a78c12bdda9e2"
server_key_version: "encryptionkey@0"
I20260619 14:36:03.496251 19426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/ts-2/wal/instance:
uuid: "3ae43cb481b940bb8be8cc303a2ad1be"
format_stamp: "Formatted at 2026-06-19 14:36:03 on dist-test-slave-70rm"
server_key: "28e3afa99f1dd344004815238aafb607"
server_key_iv: "15aa97968d86b498681a78c12bdda9e2"
server_key_version: "encryptionkey@0"
I20260619 14:36:03.500030 19426 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260619 14:36:03.502532 19441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:03.503834 19426 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:36:03.503971 19426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/ts-2/wal
uuid: "3ae43cb481b940bb8be8cc303a2ad1be"
format_stamp: "Formatted at 2026-06-19 14:36:03 on dist-test-slave-70rm"
server_key: "28e3afa99f1dd344004815238aafb607"
server_key_iv: "15aa97968d86b498681a78c12bdda9e2"
server_key_version: "encryptionkey@0"
I20260619 14:36:03.504089 19426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:36:03.515332 19426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:36:03.518464 19426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:36:03.518693 19426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:36:03.519308 19426 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:36:03.520145 19426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:36:03.520191 19426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:03.520265 19426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:36:03.520299 19426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:03.529599 19426 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.3:45747
I20260619 14:36:03.529620 19554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.3:45747 every 8 connection(s)
I20260619 14:36:03.530635 19426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
I20260619 14:36:03.537678 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 19426
I20260619 14:36:03.537833 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/ts-2/wal/instance
I20260619 14:36:03.538226 2056 external_mini_cluster.cc:1428] Setting key 02c98583b537f96e2a623f09a0859c2d
Jun 19 14:36:03 dist-test-slave-70rm krb5kdc[19051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879763, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.3@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:36:03.545403 19135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:03.532516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.3:59435 (local address 127.2.2.62:36969)
0619 14:36:03.532678 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:03.532682 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:03.533485 (+ 803us) server_negotiation.cc:407] Connection header received
0619 14:36:03.534543 (+ 1058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:03.534547 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:03.534607 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:03.534692 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:03.535984 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:03.536543 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:03.537980 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:03.538314 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:03.538429 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:03.541460 (+ 3031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:03.541463 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:03.541465 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:36:03.541492 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:03.543140 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:03.543662 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:03.543664 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:03.543666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:03.543738 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:03.544198 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:03.544201 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:03.544203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:03.544399 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:03.544477 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:03.545163 (+ 686us) server_negotiation.cc:299] Negotiation successful
0619 14:36:03.545280 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 14:36:03.546388 19557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:03.532852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:36969 (local address 127.2.2.3:59435)
0619 14:36:03.533324 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:03.533370 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 14:36:03.534317 (+ 947us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:36:03.534872 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:36:03.534879 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:36:03.535270 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:36:03.535821 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:03.535835 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:03.536686 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:03.536694 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:03.537828 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:03.537841 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:03.538422 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:03.538486 (+ 64us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:03.538684 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:03.539366 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:36:03.539382 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:36:03.541287 (+ 1905us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:36:03.543261 (+ 1974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:36:03.543267 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:36:03.543282 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:36:03.543562 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:36:03.543843 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:36:03.543848 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:36:03.543854 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:36:03.544055 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:36:03.544519 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:36:03.544525 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:36:03.544843 (+ 318us) client_negotiation.cc:769] Sending connection context
0619 14:36:03.545107 (+ 264us) client_negotiation.cc:240] Negotiation successful
0619 14:36:03.545422 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":121,"threads_started":1}
I20260619 14:36:03.547564 19555 heartbeater.cc:347] Connected to a master server at 127.2.2.62:36969
I20260619 14:36:03.547844 19555 heartbeater.cc:464] Registering TS with master...
I20260619 14:36:03.548364 19555 heartbeater.cc:511] Master 127.2.2.62:36969 requested a full tablet report, sending...
I20260619 14:36:03.549460 19096 ts_manager.cc:194] Registered new tserver with Master: 3ae43cb481b940bb8be8cc303a2ad1be (127.2.2.3:45747)
I20260619 14:36:03.550066 19096 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.2.3@KRBTEST.COM'} at 127.2.2.3:59435
I20260619 14:36:03.552485 2056 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:36:03.561188 19135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:03.554464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36672 (local address 127.2.2.62:36969)
0619 14:36:03.554588 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:03.554592 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:03.554739 (+ 147us) server_negotiation.cc:407] Connection header received
0619 14:36:03.554839 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:03.554841 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:03.554880 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:03.554949 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:03.555588 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:03.556089 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:03.556883 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:03.557090 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:03.557165 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:03.558514 (+ 1349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:03.558517 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:03.558519 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:36:03.558543 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:03.559796 (+ 1253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:03.560178 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:03.560181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:03.560182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:03.560273 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:03.560502 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:03.560504 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:03.560505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:03.560716 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:03.560815 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:03.560978 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 14:36:03.561071 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 14:36:03.564168 19096 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36672:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:36:03.567554 19096 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:36:03.582741 19566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:03.575806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.2:38833 (local address 127.0.0.1:46250)
0619 14:36:03.576646 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:03.576701 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 14:36:03.576833 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:36:03.577583 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:36:03.577591 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:36:03.577622 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:36:03.577784 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:03.577794 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:03.579324 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:03.579327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:03.580638 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:03.580649 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:03.581970 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:03.581975 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:03.582269 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:03.582305 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 14:36:03.582423 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 14:36:03.582579 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"mutex_wait_us":209,"thread_start_us":91,"threads_started":1}
I20260619 14:36:03.582839 19569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:03.576345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57358 (local address 127.2.2.1:43841)
0619 14:36:03.576664 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:03.576680 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:03.577250 (+ 570us) server_negotiation.cc:407] Connection header received
0619 14:36:03.577334 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:03.577339 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:03.577528 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:03.577668 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:36:03.578096 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:03.578905 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:03.580791 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:03.581640 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:03.581758 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:03.581903 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:03.582570 (+ 667us) server_negotiation.cc:299] Negotiation successful
0619 14:36:03.582668 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":66,"threads_started":1}
I20260619 14:36:03.583160 19567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:03.575806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.1:43841 (local address 127.0.0.1:57358)
0619 14:36:03.577061 (+ 1255us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:03.577073 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:36:03.577149 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:36:03.577823 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:36:03.577828 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:36:03.577849 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:36:03.577970 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:03.577975 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:03.579035 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:03.579041 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:03.580650 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:03.580656 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:03.581970 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:03.581975 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:03.582205 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:03.582305 (+ 100us) client_negotiation.cc:769] Sending connection context
0619 14:36:03.582423 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 14:36:03.583041 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1198,"mutex_wait_us":883,"thread_start_us":91,"threads_started":1}
I20260619 14:36:03.583372 19568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:03.576004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46250 (local address 127.2.2.2:38833)
0619 14:36:03.576873 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:03.576886 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:03.576914 (+ 28us) server_negotiation.cc:407] Connection header received
0619 14:36:03.576994 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:03.577001 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:03.577226 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:03.577377 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:36:03.577933 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:03.579185 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:03.580786 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:03.581640 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:03.581748 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:03.581903 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:03.583081 (+ 1178us) server_negotiation.cc:299] Negotiation successful
0619 14:36:03.583185 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":151,"threads_started":1}
I20260619 14:36:03.584534 19571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:03.577118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35542 (local address 127.2.2.3:45747)
0619 14:36:03.577559 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:03.577572 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:03.577609 (+ 37us) server_negotiation.cc:407] Connection header received
0619 14:36:03.577687 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:03.577691 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:03.577821 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:03.577994 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:36:03.578197 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:03.579322 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:03.581357 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:03.582940 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:03.583459 (+ 519us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:03.583793 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:03.584291 (+ 498us) server_negotiation.cc:299] Negotiation successful
0619 14:36:03.584390 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":138,"threads_started":1}
I20260619 14:36:03.584652 19570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:03.576700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.3:45747 (local address 127.0.0.1:35542)
0619 14:36:03.577419 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:03.577433 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:36:03.577524 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:36:03.577974 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:36:03.577976 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:36:03.577992 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:36:03.578091 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:03.578097 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:03.579467 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:03.579471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:03.581164 (+ 1693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:03.581178 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:03.583817 (+ 2639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:03.583825 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:03.584005 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:03.584075 (+ 70us) client_negotiation.cc:769] Sending connection context
0619 14:36:03.584250 (+ 175us) client_negotiation.cc:240] Negotiation successful
0619 14:36:03.584376 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"mutex_wait_us":181,"thread_start_us":89,"threads_started":1}
I20260619 14:36:03.586653 19355 tablet_service.cc:1467] Processing CreateTablet for tablet 472b901124d0425190d267d3d2ab7bb6 (DEFAULT_TABLE table=test-table [id=d0c22fee5dcf432cb7aa797255fabb8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:36:03.587925 19355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 472b901124d0425190d267d3d2ab7bb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:36:03.587839 19489 tablet_service.cc:1467] Processing CreateTablet for tablet 472b901124d0425190d267d3d2ab7bb6 (DEFAULT_TABLE table=test-table [id=d0c22fee5dcf432cb7aa797255fabb8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:36:03.588951 19489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 472b901124d0425190d267d3d2ab7bb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:36:03.589609 19221 tablet_service.cc:1467] Processing CreateTablet for tablet 472b901124d0425190d267d3d2ab7bb6 (DEFAULT_TABLE table=test-table [id=d0c22fee5dcf432cb7aa797255fabb8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:36:03.590657 19221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 472b901124d0425190d267d3d2ab7bb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:36:03.595594 19572 tablet_bootstrap.cc:492] T 472b901124d0425190d267d3d2ab7bb6 P bd694eb40e0e4910975c82a2496fca2d: Bootstrap starting.
I20260619 14:36:03.597119 19573 tablet_bootstrap.cc:492] T 472b901124d0425190d267d3d2ab7bb6 P 3ae43cb481b940bb8be8cc303a2ad1be: Bootstrap starting.
I20260619 14:36:03.597919 19574 tablet_bootstrap.cc:492] T 472b901124d0425190d267d3d2ab7bb6 P d2dec4eb13714c4b8e912118d35961b9: Bootstrap starting.
I20260619 14:36:03.598160 19572 tablet_bootstrap.cc:654] T 472b901124d0425190d267d3d2ab7bb6 P bd694eb40e0e4910975c82a2496fca2d: Neither blocks nor log segments found. Creating new log.
I20260619 14:36:03.599689 19572 log.cc:824] T 472b901124d0425190d267d3d2ab7bb6 P bd694eb40e0e4910975c82a2496fca2d: Log is configured to *not* fsync() on all Append() calls
I20260619 14:36:03.600241 19573 tablet_bootstrap.cc:654] T 472b901124d0425190d267d3d2ab7bb6 P 3ae43cb481b940bb8be8cc303a2ad1be: Neither blocks nor log segments found. Creating new log.
I20260619 14:36:03.600236 19574 tablet_bootstrap.cc:654] T 472b901124d0425190d267d3d2ab7bb6 P d2dec4eb13714c4b8e912118d35961b9: Neither blocks nor log segments found. Creating new log.
I20260619 14:36:03.601454 19574 log.cc:824] T 472b901124d0425190d267d3d2ab7bb6 P d2dec4eb13714c4b8e912118d35961b9: Log is configured to *not* fsync() on all Append() calls
I20260619 14:36:03.601472 19573 log.cc:824] T 472b901124d0425190d267d3d2ab7bb6 P 3ae43cb481b940bb8be8cc303a2ad1be: Log is configured to *not* fsync() on all Append() calls
I20260619 14:36:03.602190 19572 tablet_bootstrap.cc:492] T 472b901124d0425190d267d3d2ab7bb6 P bd694eb40e0e4910975c82a2496fca2d: No bootstrap required, opened a new log
I20260619 14:36:03.602406 19572 ts_tablet_manager.cc:1397] T 472b901124d0425190d267d3d2ab7bb6 P bd694eb40e0e4910975c82a2496fca2d: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260619 14:36:03.603367 19574 tablet_bootstrap.cc:492] T 472b901124d0425190d267d3d2ab7bb6 P d2dec4eb13714c4b8e912118d35961b9: No bootstrap required, opened a new log
I20260619 14:36:03.603564 19574 ts_tablet_manager.cc:1397] T 472b901124d0425190d267d3d2ab7bb6 P d2dec4eb13714c4b8e912118d35961b9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 14:36:03.603708 19573 tablet_bootstrap.cc:492] T 472b901124d0425190d267d3d2ab7bb6 P 3ae43cb481b940bb8be8cc303a2ad1be: No bootstrap required, opened a new log
I20260619 14:36:03.603888 19573 ts_tablet_manager.cc:1397] T 472b901124d0425190d267d3d2ab7bb6 P 3ae43cb481b940bb8be8cc303a2ad1be: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 14:36:03.605731 19572 raft_consensus.cc:348] T 472b901124d0425190d267d3d2ab7bb6 P bd694eb40e0e4910975c82a2496fca2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd694eb40e0e4910975c82a2496fca2d" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38833 } } peers { permanent_uuid: "d2dec4eb13714c4b8e912118d35961b9" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 43841 } } peers { permanent_uuid: "3ae43cb481b940bb8be8cc303a2ad1be" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45747 } }
I20260619 14:36:03.605948 19572 raft_consensus.cc:374] T 472b901124d0425190d267d3d2ab7bb6 P bd694eb40e0e4910975c82a2496fca2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:36:03.605988 19572 raft_consensus.cc:729] T 472b901124d0425190d267d3d2ab7bb6 P bd694eb40e0e4910975c82a2496fca2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd694eb40e0e4910975c82a2496fca2d, State: Initialized, Role: FOLLOWER
I20260619 14:36:03.606478 19572 consensus_queue.cc:260] T 472b901124d0425190d267d3d2ab7bb6 P bd694eb40e0e4910975c82a2496fca2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd694eb40e0e4910975c82a2496fca2d" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38833 } } peers { permanent_uuid: "d2dec4eb13714c4b8e912118d35961b9" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 43841 } } peers { permanent_uuid: "3ae43cb481b940bb8be8cc303a2ad1be" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45747 } }
I20260619 14:36:03.607131 19421 heartbeater.cc:503] Master 127.2.2.62:36969 was elected leader, sending a full tablet report...
I20260619 14:36:03.607508 19572 ts_tablet_manager.cc:1428] T 472b901124d0425190d267d3d2ab7bb6 P bd694eb40e0e4910975c82a2496fca2d: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:36:03.607440 19574 raft_consensus.cc:348] T 472b901124d0425190d267d3d2ab7bb6 P d2dec4eb13714c4b8e912118d35961b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd694eb40e0e4910975c82a2496fca2d" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38833 } } peers { permanent_uuid: "d2dec4eb13714c4b8e912118d35961b9" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 43841 } } peers { permanent_uuid: "3ae43cb481b940bb8be8cc303a2ad1be" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45747 } }
I20260619 14:36:03.607533 19573 raft_consensus.cc:348] T 472b901124d0425190d267d3d2ab7bb6 P 3ae43cb481b940bb8be8cc303a2ad1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd694eb40e0e4910975c82a2496fca2d" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38833 } } peers { permanent_uuid: "d2dec4eb13714c4b8e912118d35961b9" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 43841 } } peers { permanent_uuid: "3ae43cb481b940bb8be8cc303a2ad1be" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45747 } }
I20260619 14:36:03.607713 19574 raft_consensus.cc:374] T 472b901124d0425190d267d3d2ab7bb6 P d2dec4eb13714c4b8e912118d35961b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:36:03.607796 19573 raft_consensus.cc:374] T 472b901124d0425190d267d3d2ab7bb6 P 3ae43cb481b940bb8be8cc303a2ad1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:36:03.607812 19574 raft_consensus.cc:729] T 472b901124d0425190d267d3d2ab7bb6 P d2dec4eb13714c4b8e912118d35961b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2dec4eb13714c4b8e912118d35961b9, State: Initialized, Role: FOLLOWER
I20260619 14:36:03.607843 19573 raft_consensus.cc:729] T 472b901124d0425190d267d3d2ab7bb6 P 3ae43cb481b940bb8be8cc303a2ad1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ae43cb481b940bb8be8cc303a2ad1be, State: Initialized, Role: FOLLOWER
I20260619 14:36:03.608244 19573 consensus_queue.cc:260] T 472b901124d0425190d267d3d2ab7bb6 P 3ae43cb481b940bb8be8cc303a2ad1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd694eb40e0e4910975c82a2496fca2d" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38833 } } peers { permanent_uuid: "d2dec4eb13714c4b8e912118d35961b9" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 43841 } } peers { permanent_uuid: "3ae43cb481b940bb8be8cc303a2ad1be" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45747 } }
I20260619 14:36:03.608244 19574 consensus_queue.cc:260] T 472b901124d0425190d267d3d2ab7bb6 P d2dec4eb13714c4b8e912118d35961b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd694eb40e0e4910975c82a2496fca2d" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38833 } } peers { permanent_uuid: "d2dec4eb13714c4b8e912118d35961b9" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 43841 } } peers { permanent_uuid: "3ae43cb481b940bb8be8cc303a2ad1be" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45747 } }
I20260619 14:36:03.609239 19574 ts_tablet_manager.cc:1428] T 472b901124d0425190d267d3d2ab7bb6 P d2dec4eb13714c4b8e912118d35961b9: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 14:36:03.609239 19573 ts_tablet_manager.cc:1428] T 472b901124d0425190d267d3d2ab7bb6 P 3ae43cb481b940bb8be8cc303a2ad1be: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 14:36:03.609563 19287 heartbeater.cc:503] Master 127.2.2.62:36969 was elected leader, sending a full tablet report...
I20260619 14:36:03.609586 19555 heartbeater.cc:503] Master 127.2.2.62:36969 was elected leader, sending a full tablet report...
I20260619 14:36:03.707314 19580 raft_consensus.cc:482] T 472b901124d0425190d267d3d2ab7bb6 P 3ae43cb481b940bb8be8cc303a2ad1be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:36:03.707581 19580 raft_consensus.cc:504] T 472b901124d0425190d267d3d2ab7bb6 P 3ae43cb481b940bb8be8cc303a2ad1be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd694eb40e0e4910975c82a2496fca2d" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38833 } } peers { permanent_uuid: "d2dec4eb13714c4b8e912118d35961b9" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 43841 } } peers { permanent_uuid: "3ae43cb481b940bb8be8cc303a2ad1be" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45747 } }
I20260619 14:36:03.708885 19580 leader_election.cc:288] T 472b901124d0425190d267d3d2ab7bb6 P 3ae43cb481b940bb8be8cc303a2ad1be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd694eb40e0e4910975c82a2496fca2d (127.2.2.2:38833), d2dec4eb13714c4b8e912118d35961b9 (127.2.2.1:43841)
I20260619 14:36:03.714097 19569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:03.709270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.3:39905 (local address 127.2.2.1:43841)
0619 14:36:03.709411 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:03.709416 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:03.709551 (+ 135us) server_negotiation.cc:407] Connection header received
0619 14:36:03.709755 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:03.709758 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:03.709805 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:03.709884 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:36:03.710224 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:03.711005 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:03.712833 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:03.713445 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:03.713556 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:03.713620 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:03.713892 (+ 272us) server_negotiation.cc:299] Negotiation successful
0619 14:36:03.713963 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 14:36:03.714097 19568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:03.709509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.3:36925 (local address 127.2.2.2:38833)
0619 14:36:03.709647 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:03.709658 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:03.709695 (+ 37us) server_negotiation.cc:407] Connection header received
0619 14:36:03.709756 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:03.709758 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:03.709805 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:03.709884 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:36:03.710224 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:03.711005 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:03.712833 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:03.713445 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:03.713556 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:03.713620 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:03.713892 (+ 272us) server_negotiation.cc:299] Negotiation successful
0619 14:36:03.713963 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 14:36:03.714143 19557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:03.709079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.2:38833 (local address 127.2.2.3:36925)
0619 14:36:03.709425 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:03.709441 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:36:03.709538 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:36:03.709921 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:36:03.709924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:36:03.709941 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:36:03.710068 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:03.710077 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:03.711132 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:03.711135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:03.712699 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:03.712705 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:03.713577 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:03.713581 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:03.713733 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:03.713781 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 14:36:03.713857 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 14:36:03.713952 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":195}
I20260619 14:36:03.714149 19581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:03.709114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.1:43841 (local address 127.2.2.3:39905)
0619 14:36:03.709470 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:03.709484 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:36:03.709619 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:36:03.709983 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:36:03.709985 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:36:03.709997 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:36:03.710096 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:03.710104 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:03.711132 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:03.711135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:03.712693 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:03.712705 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:03.713577 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:03.713581 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:03.713715 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:03.713777 (+ 62us) client_negotiation.cc:769] Sending connection context
0619 14:36:03.713857 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 14:36:03.713952 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":208,"thread_start_us":128,"threads_started":1}
I20260619 14:36:03.714910 19241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472b901124d0425190d267d3d2ab7bb6" candidate_uuid: "3ae43cb481b940bb8be8cc303a2ad1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2dec4eb13714c4b8e912118d35961b9" is_pre_election: true
I20260619 14:36:03.714908 19375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472b901124d0425190d267d3d2ab7bb6" candidate_uuid: "3ae43cb481b940bb8be8cc303a2ad1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd694eb40e0e4910975c82a2496fca2d" is_pre_election: true
I20260619 14:36:03.715219 19241 raft_consensus.cc:2454] T 472b901124d0425190d267d3d2ab7bb6 P d2dec4eb13714c4b8e912118d35961b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ae43cb481b940bb8be8cc303a2ad1be in term 0.
I20260619 14:36:03.715233 19375 raft_consensus.cc:2454] T 472b901124d0425190d267d3d2ab7bb6 P bd694eb40e0e4910975c82a2496fca2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ae43cb481b940bb8be8cc303a2ad1be in term 0.
I20260619 14:36:03.715746 19445 leader_election.cc:302] T 472b901124d0425190d267d3d2ab7bb6 P 3ae43cb481b940bb8be8cc303a2ad1be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3ae43cb481b940bb8be8cc303a2ad1be, bd694eb40e0e4910975c82a2496fca2d, d2dec4eb13714c4b8e912118d35961b9; no voters:
I20260619 14:36:03.716043 19580 raft_consensus.cc:2781] T 472b901124d0425190d267d3d2ab7bb6 P 3ae43cb481b940bb8be8cc303a2ad1be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:36:03.716120 19580 raft_consensus.cc:482] T 472b901124d0425190d267d3d2ab7bb6 P 3ae43cb481b940bb8be8cc303a2ad1be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:36:03.716149 19580 raft_consensus.cc:3037] T 472b901124d0425190d267d3d2ab7bb6 P 3ae43cb481b940bb8be8cc303a2ad1be [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:36:03.717271 19580 raft_consensus.cc:504] T 472b901124d0425190d267d3d2ab7bb6 P 3ae43cb481b940bb8be8cc303a2ad1be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd694eb40e0e4910975c82a2496fca2d" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38833 } } peers { permanent_uuid: "d2dec4eb13714c4b8e912118d35961b9" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 43841 } } peers { permanent_uuid: "3ae43cb481b940bb8be8cc303a2ad1be" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45747 } }
I20260619 14:36:03.717706 19580 leader_election.cc:288] T 472b901124d0425190d267d3d2ab7bb6 P 3ae43cb481b940bb8be8cc303a2ad1be [CANDIDATE]: Term 1 election: Requested vote from peers bd694eb40e0e4910975c82a2496fca2d (127.2.2.2:38833), d2dec4eb13714c4b8e912118d35961b9 (127.2.2.1:43841)
I20260619 14:36:03.718173 19375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472b901124d0425190d267d3d2ab7bb6" candidate_uuid: "3ae43cb481b940bb8be8cc303a2ad1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd694eb40e0e4910975c82a2496fca2d"
I20260619 14:36:03.718196 19241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472b901124d0425190d267d3d2ab7bb6" candidate_uuid: "3ae43cb481b940bb8be8cc303a2ad1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2dec4eb13714c4b8e912118d35961b9"
I20260619 14:36:03.718343 19375 raft_consensus.cc:3037] T 472b901124d0425190d267d3d2ab7bb6 P bd694eb40e0e4910975c82a2496fca2d [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:36:03.718353 19241 raft_consensus.cc:3037] T 472b901124d0425190d267d3d2ab7bb6 P d2dec4eb13714c4b8e912118d35961b9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:36:03.719461 19375 raft_consensus.cc:2454] T 472b901124d0425190d267d3d2ab7bb6 P bd694eb40e0e4910975c82a2496fca2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ae43cb481b940bb8be8cc303a2ad1be in term 1.
I20260619 14:36:03.719475 19241 raft_consensus.cc:2454] T 472b901124d0425190d267d3d2ab7bb6 P d2dec4eb13714c4b8e912118d35961b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ae43cb481b940bb8be8cc303a2ad1be in term 1.
I20260619 14:36:03.719818 19445 leader_election.cc:302] T 472b901124d0425190d267d3d2ab7bb6 P 3ae43cb481b940bb8be8cc303a2ad1be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ae43cb481b940bb8be8cc303a2ad1be, d2dec4eb13714c4b8e912118d35961b9; no voters:
I20260619 14:36:03.720067 19580 raft_consensus.cc:2781] T 472b901124d0425190d267d3d2ab7bb6 P 3ae43cb481b940bb8be8cc303a2ad1be [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:36:03.720299 19580 raft_consensus.cc:686] T 472b901124d0425190d267d3d2ab7bb6 P 3ae43cb481b940bb8be8cc303a2ad1be [term 1 LEADER]: Becoming Leader. State: Replica: 3ae43cb481b940bb8be8cc303a2ad1be, State: Running, Role: LEADER
I20260619 14:36:03.720639 19580 consensus_queue.cc:237] T 472b901124d0425190d267d3d2ab7bb6 P 3ae43cb481b940bb8be8cc303a2ad1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd694eb40e0e4910975c82a2496fca2d" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38833 } } peers { permanent_uuid: "d2dec4eb13714c4b8e912118d35961b9" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 43841 } } peers { permanent_uuid: "3ae43cb481b940bb8be8cc303a2ad1be" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45747 } }
I20260619 14:36:03.724054 19096 catalog_manager.cc:5495] T 472b901124d0425190d267d3d2ab7bb6 P 3ae43cb481b940bb8be8cc303a2ad1be reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ae43cb481b940bb8be8cc303a2ad1be (127.2.2.3). New cstate: current_term: 1 leader_uuid: "3ae43cb481b940bb8be8cc303a2ad1be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd694eb40e0e4910975c82a2496fca2d" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2dec4eb13714c4b8e912118d35961b9" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 43841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ae43cb481b940bb8be8cc303a2ad1be" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45747 } health_report { overall_health: HEALTHY } } }
I20260619 14:36:03.747751 19571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:03.743972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35556 (local address 127.2.2.3:45747)
0619 14:36:03.744097 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:03.744100 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:03.744146 (+ 46us) server_negotiation.cc:407] Connection header received
0619 14:36:03.744288 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:03.744291 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:03.744348 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:03.744480 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:36:03.744781 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:03.745300 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:03.746259 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:03.746480 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:03.746602 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:03.746902 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:36:03.747321 (+ 419us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:36:03.747455 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:03.747514 (+ 59us) server_negotiation.cc:299] Negotiation successful
0619 14:36:03.747570 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 14:36:03.753753 19241 raft_consensus.cc:1261] T 472b901124d0425190d267d3d2ab7bb6 P d2dec4eb13714c4b8e912118d35961b9 [term 1 FOLLOWER]: Refusing update from remote peer 3ae43cb481b940bb8be8cc303a2ad1be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:36:03.753777 19375 raft_consensus.cc:1261] T 472b901124d0425190d267d3d2ab7bb6 P bd694eb40e0e4910975c82a2496fca2d [term 1 FOLLOWER]: Refusing update from remote peer 3ae43cb481b940bb8be8cc303a2ad1be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:36:03.754491 19582 consensus_queue.cc:1035] T 472b901124d0425190d267d3d2ab7bb6 P 3ae43cb481b940bb8be8cc303a2ad1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2dec4eb13714c4b8e912118d35961b9" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 43841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:36:03.754705 19580 consensus_queue.cc:1035] T 472b901124d0425190d267d3d2ab7bb6 P 3ae43cb481b940bb8be8cc303a2ad1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd694eb40e0e4910975c82a2496fca2d" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 38833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:36:03.763139 19586 mvcc.cc:205] Tried to move back new op lower bound from 7298579512327933952 to 7298579512202641408. Current Snapshot: MvccSnapshot[applied={T|T < 7298579512327933952}]
I20260619 14:36:03.765264 19587 mvcc.cc:205] Tried to move back new op lower bound from 7298579512327933952 to 7298579512202641408. Current Snapshot: MvccSnapshot[applied={T|T < 7298579512327933952 or (T in {7298579512327933952})}]
I20260619 14:36:03.770900 19095 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:36:03.771093 19095 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:36:03.773871 19095 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 418c6192aaba424b8b3abb4c29e0e98c: Sending DeleteTablet for 3 replicas of tablet 472b901124d0425190d267d3d2ab7bb6
I20260619 14:36:03.774856 19221 tablet_service.cc:1514] Processing DeleteTablet for tablet 472b901124d0425190d267d3d2ab7bb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:36:03 UTC) from {username='kudu', principal='kudu/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:57358
I20260619 14:36:03.774854 19489 tablet_service.cc:1514] Processing DeleteTablet for tablet 472b901124d0425190d267d3d2ab7bb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:36:03 UTC) from {username='kudu', principal='kudu/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:35542
I20260619 14:36:03.774878 19355 tablet_service.cc:1514] Processing DeleteTablet for tablet 472b901124d0425190d267d3d2ab7bb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:36:03 UTC) from {username='kudu', principal='kudu/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:46250
I20260619 14:36:03.775405 19595 tablet_replica.cc:330] T 472b901124d0425190d267d3d2ab7bb6 P 3ae43cb481b940bb8be8cc303a2ad1be: stopping tablet replica
I20260619 14:36:03.775655 19596 tablet_replica.cc:330] T 472b901124d0425190d267d3d2ab7bb6 P d2dec4eb13714c4b8e912118d35961b9: stopping tablet replica
I20260619 14:36:03.775771 19595 raft_consensus.cc:2229] T 472b901124d0425190d267d3d2ab7bb6 P 3ae43cb481b940bb8be8cc303a2ad1be [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:36:03.776108 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 19158
I20260619 14:36:03.776096 19596 raft_consensus.cc:2229] T 472b901124d0425190d267d3d2ab7bb6 P d2dec4eb13714c4b8e912118d35961b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:36:03.777354 19594 tablet_replica.cc:330] T 472b901124d0425190d267d3d2ab7bb6 P bd694eb40e0e4910975c82a2496fca2d: stopping tablet replica
I20260619 14:36:03.777704 19594 raft_consensus.cc:2229] T 472b901124d0425190d267d3d2ab7bb6 P bd694eb40e0e4910975c82a2496fca2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:36:03.778048 19594 raft_consensus.cc:2258] T 472b901124d0425190d267d3d2ab7bb6 P bd694eb40e0e4910975c82a2496fca2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:36:03.776225 19595 raft_consensus.cc:2258] T 472b901124d0425190d267d3d2ab7bb6 P 3ae43cb481b940bb8be8cc303a2ad1be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:36:03.779932 19595 ts_tablet_manager.cc:1905] T 472b901124d0425190d267d3d2ab7bb6 P 3ae43cb481b940bb8be8cc303a2ad1be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:36:03.779932 19594 ts_tablet_manager.cc:1905] T 472b901124d0425190d267d3d2ab7bb6 P bd694eb40e0e4910975c82a2496fca2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:36:03.782748 19594 ts_tablet_manager.cc:1918] T 472b901124d0425190d267d3d2ab7bb6 P bd694eb40e0e4910975c82a2496fca2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:36:03.782748 19595 ts_tablet_manager.cc:1918] T 472b901124d0425190d267d3d2ab7bb6 P 3ae43cb481b940bb8be8cc303a2ad1be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:36:03.782857 19594 log.cc:1196] T 472b901124d0425190d267d3d2ab7bb6 P bd694eb40e0e4910975c82a2496fca2d: Deleting WAL directory at /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/ts-1/wal/wals/472b901124d0425190d267d3d2ab7bb6
I20260619 14:36:03.782857 19595 log.cc:1196] T 472b901124d0425190d267d3d2ab7bb6 P 3ae43cb481b940bb8be8cc303a2ad1be: Deleting WAL directory at /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879514993456-2056-0/minicluster-data/ts-2/wal/wals/472b901124d0425190d267d3d2ab7bb6
I20260619 14:36:03.783206 19595 ts_tablet_manager.cc:1939] T 472b901124d0425190d267d3d2ab7bb6 P 3ae43cb481b940bb8be8cc303a2ad1be: Deleting consensus metadata
I20260619 14:36:03.783221 19594 ts_tablet_manager.cc:1939] T 472b901124d0425190d267d3d2ab7bb6 P bd694eb40e0e4910975c82a2496fca2d: Deleting consensus metadata
I20260619 14:36:03.784484 19081 catalog_manager.cc:4841] TS bd694eb40e0e4910975c82a2496fca2d (127.2.2.2:38833): tablet 472b901124d0425190d267d3d2ab7bb6 (table test-table [id=d0c22fee5dcf432cb7aa797255fabb8f]) successfully deleted
I20260619 14:36:03.784694 19081 catalog_manager.cc:4841] TS 3ae43cb481b940bb8be8cc303a2ad1be (127.2.2.3:45747): tablet 472b901124d0425190d267d3d2ab7bb6 (table test-table [id=d0c22fee5dcf432cb7aa797255fabb8f]) successfully deleted
W20260619 14:36:03.785040 19083 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.2.1:43841 (error 108)
I20260619 14:36:03.785241 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 19292
I20260619 14:36:03.787055 19570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:03.786739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.1:43841 (local address 127.0.0.1:57372)
0619 14:36:03.786882 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:03.786951 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.1:43841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260619 14:36:03.787226 19083 catalog_manager.cc:4584] TS d2dec4eb13714c4b8e912118d35961b9 (127.2.2.1:43841): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 472b901124d0425190d267d3d2ab7bb6: Network error: Client connection negotiation failed: client connection to 127.2.2.1:43841: connect: Connection refused (error 111)
I20260619 14:36:03.792163 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 19426
I20260619 14:36:03.797729 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 19064
2026-06-19T14:36:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2323 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-19T14:36:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:36:03Z Disabled control of system clock
I20260619 14:36:03.827739 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:32935
--webserver_interface=127.2.2.62
--webserver_port=0
--builtin_ntp_servers=127.2.2.20:45127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.2.62:32935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 14:36:03.936249 19601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:36:03.936512 19601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:36:03.936547 19601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:36:03.940467 19601 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:36:03.940524 19601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:36:03.940541 19601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:36:03.940557 19601 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:36:03.940572 19601 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:36:03.945756 19601 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:45127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.2.62:32935
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:32935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.2.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:36:03.946815 19601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:36:03.948129 19601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:36:03.956804 19601 server_base.cc:1034] running on GCE node
W20260619 14:36:03.956847 19607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:03.956777 19606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:03.956812 19609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:03.957532 19601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:36:03.958739 19601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:36:03.959908 19601 hybrid_clock.cc:648] HybridClock initialized: now 1781879763959891 us; error 28 us; skew 500 ppm
I20260619 14:36:03.961730 19601 webserver.cc:460] Webserver started at http://127.2.2.62:42149/ using document root <none> and password file <none>
I20260619 14:36:03.962379 19601 fs_manager.cc:359] Metadata directory not provided
I20260619 14:36:03.962442 19601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:36:03.962699 19601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:36:03.964713 19601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/master-0/data/instance:
uuid: "b90d0ae82e5a4bc69f05c8bfc4b47258"
format_stamp: "Formatted at 2026-06-19 14:36:03 on dist-test-slave-70rm"
server_key: "5e1218ba8c0e146c5622bb3779549a84"
server_key_iv: "5f035164b2cf05f6e0160f39217853b4"
server_key_version: "encryptionkey@0"
I20260619 14:36:03.965274 19601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/master-0/wal/instance:
uuid: "b90d0ae82e5a4bc69f05c8bfc4b47258"
format_stamp: "Formatted at 2026-06-19 14:36:03 on dist-test-slave-70rm"
server_key: "5e1218ba8c0e146c5622bb3779549a84"
server_key_iv: "5f035164b2cf05f6e0160f39217853b4"
server_key_version: "encryptionkey@0"
I20260619 14:36:03.968876 19601 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260619 14:36:03.971284 19615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:03.972469 19601 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:36:03.972595 19601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/master-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/master-0/wal
uuid: "b90d0ae82e5a4bc69f05c8bfc4b47258"
format_stamp: "Formatted at 2026-06-19 14:36:03 on dist-test-slave-70rm"
server_key: "5e1218ba8c0e146c5622bb3779549a84"
server_key_iv: "5f035164b2cf05f6e0160f39217853b4"
server_key_version: "encryptionkey@0"
I20260619 14:36:03.972705 19601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:36:03.989308 19601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:36:03.989959 19601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:36:03.990108 19601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:36:03.996987 19601 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.62:32935
I20260619 14:36:03.996994 19667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.62:32935 every 8 connection(s)
I20260619 14:36:03.997929 19601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
I20260619 14:36:04.001025 19668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:36:04.003439 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 19601
I20260619 14:36:04.003538 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/master-0/wal/instance
I20260619 14:36:04.003811 2056 external_mini_cluster.cc:1428] Setting key 74383290a6243e467c08911d537eb0ae
I20260619 14:36:04.006825 19668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b90d0ae82e5a4bc69f05c8bfc4b47258: Bootstrap starting.
I20260619 14:36:04.009248 19668 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b90d0ae82e5a4bc69f05c8bfc4b47258: Neither blocks nor log segments found. Creating new log.
I20260619 14:36:04.010074 19668 log.cc:824] T 00000000000000000000000000000000 P b90d0ae82e5a4bc69f05c8bfc4b47258: Log is configured to *not* fsync() on all Append() calls
I20260619 14:36:04.011947 19668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b90d0ae82e5a4bc69f05c8bfc4b47258: No bootstrap required, opened a new log
I20260619 14:36:04.012993 19671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:04.005235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40342 (local address 127.2.2.62:32935)
0619 14:36:04.005840 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:04.005853 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:04.005889 (+ 36us) server_negotiation.cc:407] Connection header received
0619 14:36:04.006553 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:04.006603 (+ 50us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:04.007036 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:04.007313 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:04.007648 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:04.008578 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:04.009774 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:04.010333 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:04.010447 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:04.011334 (+ 887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:04.011340 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:04.011363 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:36:04.011730 (+ 367us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:04.011795 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:36:04.011814 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:04.011920 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:04.012067 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 14:36:04.012325 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":127,"threads_started":1}
I20260619 14:36:04.015529 19668 raft_consensus.cc:348] T 00000000000000000000000000000000 P b90d0ae82e5a4bc69f05c8bfc4b47258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90d0ae82e5a4bc69f05c8bfc4b47258" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 32935 } }
I20260619 14:36:04.015759 19668 raft_consensus.cc:374] T 00000000000000000000000000000000 P b90d0ae82e5a4bc69f05c8bfc4b47258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:36:04.015836 19668 raft_consensus.cc:729] T 00000000000000000000000000000000 P b90d0ae82e5a4bc69f05c8bfc4b47258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b90d0ae82e5a4bc69f05c8bfc4b47258, State: Initialized, Role: FOLLOWER
I20260619 14:36:04.016278 19668 consensus_queue.cc:260] T 00000000000000000000000000000000 P b90d0ae82e5a4bc69f05c8bfc4b47258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90d0ae82e5a4bc69f05c8bfc4b47258" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 32935 } }
I20260619 14:36:04.016423 19668 raft_consensus.cc:388] T 00000000000000000000000000000000 P b90d0ae82e5a4bc69f05c8bfc4b47258 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:36:04.016505 19668 raft_consensus.cc:482] T 00000000000000000000000000000000 P b90d0ae82e5a4bc69f05c8bfc4b47258 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:36:04.016601 19668 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b90d0ae82e5a4bc69f05c8bfc4b47258 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:36:04.017480 19668 raft_consensus.cc:504] T 00000000000000000000000000000000 P b90d0ae82e5a4bc69f05c8bfc4b47258 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90d0ae82e5a4bc69f05c8bfc4b47258" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 32935 } }
I20260619 14:36:04.017809 19668 leader_election.cc:302] T 00000000000000000000000000000000 P b90d0ae82e5a4bc69f05c8bfc4b47258 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b90d0ae82e5a4bc69f05c8bfc4b47258; no voters:
I20260619 14:36:04.018165 19668 leader_election.cc:288] T 00000000000000000000000000000000 P b90d0ae82e5a4bc69f05c8bfc4b47258 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:36:04.018267 19673 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b90d0ae82e5a4bc69f05c8bfc4b47258 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:36:04.018541 19673 raft_consensus.cc:686] T 00000000000000000000000000000000 P b90d0ae82e5a4bc69f05c8bfc4b47258 [term 1 LEADER]: Becoming Leader. State: Replica: b90d0ae82e5a4bc69f05c8bfc4b47258, State: Running, Role: LEADER
I20260619 14:36:04.018854 19673 consensus_queue.cc:237] T 00000000000000000000000000000000 P b90d0ae82e5a4bc69f05c8bfc4b47258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90d0ae82e5a4bc69f05c8bfc4b47258" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 32935 } }
I20260619 14:36:04.019467 19668 sys_catalog.cc:564] T 00000000000000000000000000000000 P b90d0ae82e5a4bc69f05c8bfc4b47258 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:36:04.020466 19675 sys_catalog.cc:455] T 00000000000000000000000000000000 P b90d0ae82e5a4bc69f05c8bfc4b47258 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b90d0ae82e5a4bc69f05c8bfc4b47258. Latest consensus state: current_term: 1 leader_uuid: "b90d0ae82e5a4bc69f05c8bfc4b47258" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90d0ae82e5a4bc69f05c8bfc4b47258" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 32935 } } }
I20260619 14:36:04.020583 19675 sys_catalog.cc:458] T 00000000000000000000000000000000 P b90d0ae82e5a4bc69f05c8bfc4b47258 [sys.catalog]: This master's current role is: LEADER
I20260619 14:36:04.021164 19674 sys_catalog.cc:455] T 00000000000000000000000000000000 P b90d0ae82e5a4bc69f05c8bfc4b47258 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b90d0ae82e5a4bc69f05c8bfc4b47258" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90d0ae82e5a4bc69f05c8bfc4b47258" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 32935 } } }
I20260619 14:36:04.021274 19674 sys_catalog.cc:458] T 00000000000000000000000000000000 P b90d0ae82e5a4bc69f05c8bfc4b47258 [sys.catalog]: This master's current role is: LEADER
I20260619 14:36:04.021762 19682 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:36:04.024284 19682 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:36:04.029668 19682 catalog_manager.cc:1347] Generated new cluster ID: d9819acff7b64c3bb5e9e5cee1f7ef81
I20260619 14:36:04.029752 19682 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:36:04.062228 19682 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:36:04.063192 19682 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:36:04.072224 19682 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b90d0ae82e5a4bc69f05c8bfc4b47258: Generated new TSK 0
I20260619 14:36:04.072953 19682 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:36:04.077273 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.1:0
--local_ip_for_outbound_sockets=127.2.2.1
--webserver_interface=127.2.2.1
--webserver_port=0
--tserver_master_addrs=127.2.2.62:32935
--builtin_ntp_servers=127.2.2.20:45127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 14:36:04.190999 19692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:36:04.191290 19692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:36:04.191332 19692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:36:04.195524 19692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:36:04.195586 19692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:36:04.195672 19692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.1
I20260619 14:36:04.200698 19692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:45127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.2.1
--webserver_port=0
--tserver_master_addrs=127.2.2.62:32935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.1
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:36:04.201823 19692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:36:04.203290 19692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:36:04.212689 19700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:04.212639 19698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:04.212663 19697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:04.212915 19692 server_base.cc:1034] running on GCE node
I20260619 14:36:04.213476 19692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:36:04.214107 19692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:36:04.215341 19692 hybrid_clock.cc:648] HybridClock initialized: now 1781879764215311 us; error 51 us; skew 500 ppm
I20260619 14:36:04.217278 19692 webserver.cc:460] Webserver started at http://127.2.2.1:44079/ using document root <none> and password file <none>
I20260619 14:36:04.217869 19692 fs_manager.cc:359] Metadata directory not provided
I20260619 14:36:04.217965 19692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:36:04.218252 19692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:36:04.220538 19692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/ts-0/data/instance:
uuid: "fbf33decd3e4423b86d9cc954f957270"
format_stamp: "Formatted at 2026-06-19 14:36:04 on dist-test-slave-70rm"
server_key: "58338c9913ac0bd682f4d602a2fde959"
server_key_iv: "6203a1093c578951d4d82bb8b457352e"
server_key_version: "encryptionkey@0"
I20260619 14:36:04.221314 19692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/ts-0/wal/instance:
uuid: "fbf33decd3e4423b86d9cc954f957270"
format_stamp: "Formatted at 2026-06-19 14:36:04 on dist-test-slave-70rm"
server_key: "58338c9913ac0bd682f4d602a2fde959"
server_key_iv: "6203a1093c578951d4d82bb8b457352e"
server_key_version: "encryptionkey@0"
I20260619 14:36:04.224988 19692 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.005s
I20260619 14:36:04.227478 19706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:04.228749 19692 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:36:04.228896 19692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/ts-0/wal
uuid: "fbf33decd3e4423b86d9cc954f957270"
format_stamp: "Formatted at 2026-06-19 14:36:04 on dist-test-slave-70rm"
server_key: "58338c9913ac0bd682f4d602a2fde959"
server_key_iv: "6203a1093c578951d4d82bb8b457352e"
server_key_version: "encryptionkey@0"
I20260619 14:36:04.229018 19692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:36:04.251892 19692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:36:04.252648 19692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:36:04.252866 19692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:36:04.253484 19692 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:36:04.254504 19692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:36:04.254577 19692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:04.254650 19692 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:36:04.254693 19692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:04.263212 19692 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.1:33561
I20260619 14:36:04.263232 19819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.1:33561 every 8 connection(s)
I20260619 14:36:04.264204 19692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
I20260619 14:36:04.273743 19671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:04.266203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:41295 (local address 127.2.2.62:32935)
0619 14:36:04.266346 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:04.266349 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:04.267487 (+ 1138us) server_negotiation.cc:407] Connection header received
0619 14:36:04.268921 (+ 1434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:04.268924 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:04.268980 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:04.269089 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:04.270005 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:04.270607 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:04.271767 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:04.271997 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:04.272111 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:04.273224 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:04.273227 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:04.273229 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:36:04.273243 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:04.273265 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:36:04.273272 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:04.273351 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:04.273514 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 14:36:04.273580 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 14:36:04.273844 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 19692
I20260619 14:36:04.273967 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/ts-0/wal/instance
I20260619 14:36:04.274293 2056 external_mini_cluster.cc:1428] Setting key 7219a6b3398621fca8defc2888d7c373
I20260619 14:36:04.274791 19822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:04.266651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:32935 (local address 127.2.2.1:41295)
0619 14:36:04.267330 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:04.267372 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 14:36:04.268678 (+ 1306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:36:04.269280 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:36:04.269287 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:36:04.269374 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:36:04.269863 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:04.269875 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:04.270736 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:04.270739 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:04.271636 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:04.271645 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:04.272098 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:04.272159 (+ 61us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:04.272336 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:04.272945 (+ 609us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:36:04.272947 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:36:04.272989 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:36:04.273053 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:36:04.273352 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:36:04.273365 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:36:04.273372 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 14:36:04.273505 (+ 133us) client_negotiation.cc:240] Negotiation successful
0619 14:36:04.273823 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":108,"threads_started":1}
I20260619 14:36:04.276194 19820 heartbeater.cc:347] Connected to a master server at 127.2.2.62:32935
I20260619 14:36:04.276290 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.2:0
--local_ip_for_outbound_sockets=127.2.2.2
--webserver_interface=127.2.2.2
--webserver_port=0
--tserver_master_addrs=127.2.2.62:32935
--builtin_ntp_servers=127.2.2.20:45127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 14:36:04.276458 19820 heartbeater.cc:464] Registering TS with master...
I20260619 14:36:04.277110 19820 heartbeater.cc:511] Master 127.2.2.62:32935 requested a full tablet report, sending...
I20260619 14:36:04.278792 19631 ts_manager.cc:194] Registered new tserver with Master: fbf33decd3e4423b86d9cc954f957270 (127.2.2.1:33561)
I20260619 14:36:04.279917 19631 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.2.1:41295
W20260619 14:36:04.390834 19823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:36:04.391155 19823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:36:04.391196 19823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:36:04.395380 19823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:36:04.395448 19823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:36:04.395506 19823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.2
I20260619 14:36:04.400352 19823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:45127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.2.2
--webserver_port=0
--tserver_master_addrs=127.2.2.62:32935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.2
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:36:04.401417 19823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:36:04.402777 19823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:36:04.412380 19828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:04.412453 19829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:04.412446 19831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:04.412639 19823 server_base.cc:1034] running on GCE node
I20260619 14:36:04.413108 19823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:36:04.413740 19823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:36:04.414928 19823 hybrid_clock.cc:648] HybridClock initialized: now 1781879764414897 us; error 54 us; skew 500 ppm
I20260619 14:36:04.416843 19823 webserver.cc:460] Webserver started at http://127.2.2.2:33927/ using document root <none> and password file <none>
I20260619 14:36:04.417433 19823 fs_manager.cc:359] Metadata directory not provided
I20260619 14:36:04.417526 19823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:36:04.417769 19823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:36:04.419989 19823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/ts-1/data/instance:
uuid: "2b08d7c01e224bf2aaefad82a6e549db"
format_stamp: "Formatted at 2026-06-19 14:36:04 on dist-test-slave-70rm"
server_key: "d5bd2a38507ac25f938ea5f3688fde66"
server_key_iv: "6598624d975ef928f8b9cf8083583ffb"
server_key_version: "encryptionkey@0"
I20260619 14:36:04.420621 19823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/ts-1/wal/instance:
uuid: "2b08d7c01e224bf2aaefad82a6e549db"
format_stamp: "Formatted at 2026-06-19 14:36:04 on dist-test-slave-70rm"
server_key: "d5bd2a38507ac25f938ea5f3688fde66"
server_key_iv: "6598624d975ef928f8b9cf8083583ffb"
server_key_version: "encryptionkey@0"
I20260619 14:36:04.424427 19823 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260619 14:36:04.426952 19837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:04.428221 19823 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:36:04.428546 19823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/ts-1/wal
uuid: "2b08d7c01e224bf2aaefad82a6e549db"
format_stamp: "Formatted at 2026-06-19 14:36:04 on dist-test-slave-70rm"
server_key: "d5bd2a38507ac25f938ea5f3688fde66"
server_key_iv: "6598624d975ef928f8b9cf8083583ffb"
server_key_version: "encryptionkey@0"
I20260619 14:36:04.428696 19823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:36:04.451947 19823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:36:04.452711 19823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:36:04.452919 19823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:36:04.453542 19823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:36:04.454586 19823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:36:04.454656 19823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:04.454730 19823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:36:04.454771 19823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:04.463774 19823 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.2:39203
I20260619 14:36:04.463795 19950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.2:39203 every 8 connection(s)
I20260619 14:36:04.464727 19823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
I20260619 14:36:04.465167 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 19823
I20260619 14:36:04.465276 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/ts-1/wal/instance
I20260619 14:36:04.465598 2056 external_mini_cluster.cc:1428] Setting key ff9700127a50e875b9a48fd942a5f44c
I20260619 14:36:04.467973 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.3:0
--local_ip_for_outbound_sockets=127.2.2.3
--webserver_interface=127.2.2.3
--webserver_port=0
--tserver_master_addrs=127.2.2.62:32935
--builtin_ntp_servers=127.2.2.20:45127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 14:36:04.474947 19671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:04.466647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:59721 (local address 127.2.2.62:32935)
0619 14:36:04.466793 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:04.466798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:04.468120 (+ 1322us) server_negotiation.cc:407] Connection header received
0619 14:36:04.470069 (+ 1949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:04.470072 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:04.470148 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:04.470231 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:04.471172 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:04.471715 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:04.472885 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:04.473114 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:04.473228 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:04.474412 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:04.474416 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:04.474419 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:36:04.474439 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:04.474459 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:36:04.474467 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:04.474553 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:04.474722 (+ 169us) server_negotiation.cc:299] Negotiation successful
0619 14:36:04.474797 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 14:36:04.475814 19953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:04.467164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:32935 (local address 127.2.2.2:59721)
0619 14:36:04.467936 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:04.467995 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 14:36:04.469860 (+ 1865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:36:04.470415 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:36:04.470424 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:36:04.470538 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:36:04.471026 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:04.471039 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:04.471856 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:04.471859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:04.472757 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:04.472766 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:04.473215 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:04.473280 (+ 65us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:04.473450 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:04.474084 (+ 634us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:36:04.474086 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:36:04.474172 (+ 86us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:36:04.474255 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:36:04.474554 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:36:04.474565 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:36:04.474573 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 14:36:04.474710 (+ 137us) client_negotiation.cc:240] Negotiation successful
0619 14:36:04.474998 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":117,"threads_started":1}
I20260619 14:36:04.477011 19951 heartbeater.cc:347] Connected to a master server at 127.2.2.62:32935
I20260619 14:36:04.477310 19951 heartbeater.cc:464] Registering TS with master...
I20260619 14:36:04.477862 19951 heartbeater.cc:511] Master 127.2.2.62:32935 requested a full tablet report, sending...
I20260619 14:36:04.479039 19631 ts_manager.cc:194] Registered new tserver with Master: 2b08d7c01e224bf2aaefad82a6e549db (127.2.2.2:39203)
I20260619 14:36:04.479809 19631 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.2.2:59721
W20260619 14:36:04.582105 19954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:36:04.582448 19954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:36:04.582489 19954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:36:04.586509 19954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:36:04.586589 19954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:36:04.586647 19954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.3
I20260619 14:36:04.591434 19954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:45127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.2.3
--webserver_port=0
--tserver_master_addrs=127.2.2.62:32935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.3
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:36:04.592494 19954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:36:04.593835 19954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:36:04.603296 19960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:04.603283 19959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:04.603608 19962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:04.603598 19954 server_base.cc:1034] running on GCE node
I20260619 14:36:04.604014 19954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:36:04.604593 19954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:36:04.605765 19954 hybrid_clock.cc:648] HybridClock initialized: now 1781879764605752 us; error 31 us; skew 500 ppm
I20260619 14:36:04.607604 19954 webserver.cc:460] Webserver started at http://127.2.2.3:42827/ using document root <none> and password file <none>
I20260619 14:36:04.608191 19954 fs_manager.cc:359] Metadata directory not provided
I20260619 14:36:04.608280 19954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:36:04.608512 19954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:36:04.610595 19954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/ts-2/data/instance:
uuid: "28101784cc0f42a68e25f371f8744d5b"
format_stamp: "Formatted at 2026-06-19 14:36:04 on dist-test-slave-70rm"
server_key: "100628b41d18462de700c40c92465a43"
server_key_iv: "5808be7189cb9fa5d56210a888a2398f"
server_key_version: "encryptionkey@0"
I20260619 14:36:04.611169 19954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/ts-2/wal/instance:
uuid: "28101784cc0f42a68e25f371f8744d5b"
format_stamp: "Formatted at 2026-06-19 14:36:04 on dist-test-slave-70rm"
server_key: "100628b41d18462de700c40c92465a43"
server_key_iv: "5808be7189cb9fa5d56210a888a2398f"
server_key_version: "encryptionkey@0"
I20260619 14:36:04.614833 19954 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:36:04.617383 19968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:04.618654 19954 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:36:04.618783 19954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/ts-2/wal
uuid: "28101784cc0f42a68e25f371f8744d5b"
format_stamp: "Formatted at 2026-06-19 14:36:04 on dist-test-slave-70rm"
server_key: "100628b41d18462de700c40c92465a43"
server_key_iv: "5808be7189cb9fa5d56210a888a2398f"
server_key_version: "encryptionkey@0"
I20260619 14:36:04.618896 19954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:36:04.630363 19954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:36:04.631085 19954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:36:04.631283 19954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:36:04.631984 19954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:36:04.633106 19954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:36:04.633154 19954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:04.633219 19954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:36:04.633253 19954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:04.642128 19954 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.3:36081
I20260619 14:36:04.642165 20081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.3:36081 every 8 connection(s)
I20260619 14:36:04.643085 19954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
I20260619 14:36:04.644397 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 19954
I20260619 14:36:04.644507 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/ts-2/wal/instance
I20260619 14:36:04.644774 2056 external_mini_cluster.cc:1428] Setting key 3a2c029e37326c07cd2aee26b86c7069
I20260619 14:36:04.652420 19671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:04.645019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.3:33171 (local address 127.2.2.62:32935)
0619 14:36:04.645181 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:04.645187 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:04.646518 (+ 1331us) server_negotiation.cc:407] Connection header received
0619 14:36:04.647865 (+ 1347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:04.647868 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:04.647912 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:04.647979 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:04.648954 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:04.649462 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:04.650680 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:04.650909 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:04.650988 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:04.652019 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:04.652022 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:04.652024 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:36:04.652038 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:04.652056 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:36:04.652064 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:04.652118 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:04.652251 (+ 133us) server_negotiation.cc:299] Negotiation successful
0619 14:36:04.652304 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 14:36:04.653293 20084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:04.645552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:32935 (local address 127.2.2.3:33171)
0619 14:36:04.646350 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:04.646405 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 14:36:04.647678 (+ 1273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:36:04.648193 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:36:04.648200 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:36:04.648306 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:36:04.648821 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:04.648834 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:04.649570 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:04.649572 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:04.650572 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:04.650582 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:04.650991 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:04.651038 (+ 47us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:04.651180 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:04.651774 (+ 594us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:36:04.651775 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:36:04.651816 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:36:04.651881 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:36:04.652130 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:36:04.652136 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:36:04.652140 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 14:36:04.652249 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 14:36:04.652477 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":142,"threads_started":1}
I20260619 14:36:04.654392 20082 heartbeater.cc:347] Connected to a master server at 127.2.2.62:32935
I20260619 14:36:04.654628 20082 heartbeater.cc:464] Registering TS with master...
I20260619 14:36:04.655144 20082 heartbeater.cc:511] Master 127.2.2.62:32935 requested a full tablet report, sending...
I20260619 14:36:04.656121 19631 ts_manager.cc:194] Registered new tserver with Master: 28101784cc0f42a68e25f371f8744d5b (127.2.2.3:36081)
I20260619 14:36:04.656697 19631 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.2.3:33171
I20260619 14:36:04.658779 2056 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:36:04.664129 19671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:04.660659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40352 (local address 127.2.2.62:32935)
0619 14:36:04.660807 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:04.660811 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:04.660992 (+ 181us) server_negotiation.cc:407] Connection header received
0619 14:36:04.661079 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:04.661081 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:04.661120 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:04.661184 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:04.661431 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:04.661928 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:04.662761 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:04.662964 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:04.663042 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:04.663729 (+ 687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:04.663732 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:04.663734 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:36:04.663755 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:04.663773 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:36:04.663781 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:04.663836 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:04.663905 (+ 69us) server_negotiation.cc:299] Negotiation successful
0619 14:36:04.663952 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 14:36:04.666872 19631 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40352:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:36:04.669782 19631 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:36:04.684355 20093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:04.678305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.3:36081 (local address 127.0.0.1:52964)
0619 14:36:04.678776 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:04.678829 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 14:36:04.678967 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:36:04.679690 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:36:04.679697 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:36:04.679730 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:36:04.679880 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:04.679888 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:04.681468 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:04.681472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:04.682854 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:04.682864 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:04.683731 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:04.683735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:04.683852 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:04.683895 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 14:36:04.684029 (+ 134us) client_negotiation.cc:240] Negotiation successful
0619 14:36:04.684171 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":32,"thread_start_us":181,"threads_started":1}
I20260619 14:36:04.684357 20094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:04.678486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52964 (local address 127.2.2.3:36081)
0619 14:36:04.679140 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:04.679153 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:04.679179 (+ 26us) server_negotiation.cc:407] Connection header received
0619 14:36:04.679271 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:04.679277 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:04.679509 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:04.680050 (+ 541us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:36:04.680198 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:04.681322 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:04.683000 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:04.683583 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:04.683719 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:04.683939 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:04.684060 (+ 121us) server_negotiation.cc:299] Negotiation successful
0619 14:36:04.684171 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":185,"threads_started":1}
I20260619 14:36:04.685284 20097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:04.679366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39762 (local address 127.2.2.2:39203)
0619 14:36:04.680215 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:04.680230 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:04.680259 (+ 29us) server_negotiation.cc:407] Connection header received
0619 14:36:04.680347 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:04.680350 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:04.680523 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:04.680696 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:36:04.681091 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:04.682168 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:04.683817 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:04.684626 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:04.684735 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:04.684909 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:04.685000 (+ 91us) server_negotiation.cc:299] Negotiation successful
0619 14:36:04.685090 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":742,"thread_start_us":131,"threads_started":1}
I20260619 14:36:04.685385 20096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:04.679202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.1:33561 (local address 127.0.0.1:60112)
0619 14:36:04.680075 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:04.680089 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:36:04.680191 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:36:04.680646 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:36:04.680650 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:36:04.680675 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:36:04.680785 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:04.680793 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:04.682025 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:04.682028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:04.683668 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:04.683679 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:04.684978 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:04.684981 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:04.685069 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:04.685092 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 14:36:04.685164 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 14:36:04.685237 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":808,"mutex_wait_us":83,"thread_start_us":101,"threads_started":1}
I20260619 14:36:04.685854 20098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:04.679673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60112 (local address 127.2.2.1:33561)
0619 14:36:04.680028 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:04.680039 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:04.680188 (+ 149us) server_negotiation.cc:407] Connection header received
0619 14:36:04.680301 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:04.680306 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:04.680489 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:04.680639 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:36:04.680908 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:04.681901 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:04.683817 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:04.684626 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:04.684746 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:04.684909 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:04.685560 (+ 651us) server_negotiation.cc:299] Negotiation successful
0619 14:36:04.685670 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":130,"threads_started":1}
I20260619 14:36:04.685940 20095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:04.679022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.2:39203 (local address 127.0.0.1:39762)
0619 14:36:04.679815 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:04.679833 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:36:04.679935 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:36:04.680839 (+ 904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:36:04.680844 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:36:04.680865 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:36:04.680966 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:04.680971 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:04.682291 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:04.682293 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:04.683673 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:04.683679 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:04.684758 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:04.684762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:04.684867 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:04.684894 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 14:36:04.684975 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 14:36:04.685056 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"mutex_wait_us":130,"thread_start_us":89,"threads_started":1}
I20260619 14:36:04.688390 20016 tablet_service.cc:1467] Processing CreateTablet for tablet bb0502fac32c47778f41ba3b6793ce85 (DEFAULT_TABLE table=test-table [id=25666854fe4e4537ac89c6a71f49c8e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:36:04.688719 19885 tablet_service.cc:1467] Processing CreateTablet for tablet bb0502fac32c47778f41ba3b6793ce85 (DEFAULT_TABLE table=test-table [id=25666854fe4e4537ac89c6a71f49c8e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:36:04.689692 19885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb0502fac32c47778f41ba3b6793ce85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:36:04.689692 20016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb0502fac32c47778f41ba3b6793ce85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:36:04.691519 19754 tablet_service.cc:1467] Processing CreateTablet for tablet bb0502fac32c47778f41ba3b6793ce85 (DEFAULT_TABLE table=test-table [id=25666854fe4e4537ac89c6a71f49c8e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:36:04.692371 19754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb0502fac32c47778f41ba3b6793ce85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:36:04.697654 20100 tablet_bootstrap.cc:492] T bb0502fac32c47778f41ba3b6793ce85 P 28101784cc0f42a68e25f371f8744d5b: Bootstrap starting.
I20260619 14:36:04.698108 20099 tablet_bootstrap.cc:492] T bb0502fac32c47778f41ba3b6793ce85 P 2b08d7c01e224bf2aaefad82a6e549db: Bootstrap starting.
I20260619 14:36:04.699385 20101 tablet_bootstrap.cc:492] T bb0502fac32c47778f41ba3b6793ce85 P fbf33decd3e4423b86d9cc954f957270: Bootstrap starting.
I20260619 14:36:04.700654 20099 tablet_bootstrap.cc:654] T bb0502fac32c47778f41ba3b6793ce85 P 2b08d7c01e224bf2aaefad82a6e549db: Neither blocks nor log segments found. Creating new log.
I20260619 14:36:04.701507 20101 tablet_bootstrap.cc:654] T bb0502fac32c47778f41ba3b6793ce85 P fbf33decd3e4423b86d9cc954f957270: Neither blocks nor log segments found. Creating new log.
I20260619 14:36:04.701535 20099 log.cc:824] T bb0502fac32c47778f41ba3b6793ce85 P 2b08d7c01e224bf2aaefad82a6e549db: Log is configured to *not* fsync() on all Append() calls
I20260619 14:36:04.702347 20101 log.cc:824] T bb0502fac32c47778f41ba3b6793ce85 P fbf33decd3e4423b86d9cc954f957270: Log is configured to *not* fsync() on all Append() calls
I20260619 14:36:04.703617 20099 tablet_bootstrap.cc:492] T bb0502fac32c47778f41ba3b6793ce85 P 2b08d7c01e224bf2aaefad82a6e549db: No bootstrap required, opened a new log
I20260619 14:36:04.703838 20099 ts_tablet_manager.cc:1397] T bb0502fac32c47778f41ba3b6793ce85 P 2b08d7c01e224bf2aaefad82a6e549db: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260619 14:36:04.704102 20100 tablet_bootstrap.cc:654] T bb0502fac32c47778f41ba3b6793ce85 P 28101784cc0f42a68e25f371f8744d5b: Neither blocks nor log segments found. Creating new log.
I20260619 14:36:04.704368 20101 tablet_bootstrap.cc:492] T bb0502fac32c47778f41ba3b6793ce85 P fbf33decd3e4423b86d9cc954f957270: No bootstrap required, opened a new log
I20260619 14:36:04.704546 20101 ts_tablet_manager.cc:1397] T bb0502fac32c47778f41ba3b6793ce85 P fbf33decd3e4423b86d9cc954f957270: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:36:04.704942 20100 log.cc:824] T bb0502fac32c47778f41ba3b6793ce85 P 28101784cc0f42a68e25f371f8744d5b: Log is configured to *not* fsync() on all Append() calls
I20260619 14:36:04.706915 20100 tablet_bootstrap.cc:492] T bb0502fac32c47778f41ba3b6793ce85 P 28101784cc0f42a68e25f371f8744d5b: No bootstrap required, opened a new log
I20260619 14:36:04.707146 20100 ts_tablet_manager.cc:1397] T bb0502fac32c47778f41ba3b6793ce85 P 28101784cc0f42a68e25f371f8744d5b: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260619 14:36:04.707736 20099 raft_consensus.cc:348] T bb0502fac32c47778f41ba3b6793ce85 P 2b08d7c01e224bf2aaefad82a6e549db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28101784cc0f42a68e25f371f8744d5b" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 36081 } } peers { permanent_uuid: "fbf33decd3e4423b86d9cc954f957270" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 33561 } } peers { permanent_uuid: "2b08d7c01e224bf2aaefad82a6e549db" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 39203 } }
I20260619 14:36:04.707983 20099 raft_consensus.cc:374] T bb0502fac32c47778f41ba3b6793ce85 P 2b08d7c01e224bf2aaefad82a6e549db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:36:04.708057 20099 raft_consensus.cc:729] T bb0502fac32c47778f41ba3b6793ce85 P 2b08d7c01e224bf2aaefad82a6e549db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b08d7c01e224bf2aaefad82a6e549db, State: Initialized, Role: FOLLOWER
I20260619 14:36:04.708500 20099 consensus_queue.cc:260] T bb0502fac32c47778f41ba3b6793ce85 P 2b08d7c01e224bf2aaefad82a6e549db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28101784cc0f42a68e25f371f8744d5b" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 36081 } } peers { permanent_uuid: "fbf33decd3e4423b86d9cc954f957270" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 33561 } } peers { permanent_uuid: "2b08d7c01e224bf2aaefad82a6e549db" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 39203 } }
I20260619 14:36:04.708698 20101 raft_consensus.cc:348] T bb0502fac32c47778f41ba3b6793ce85 P fbf33decd3e4423b86d9cc954f957270 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28101784cc0f42a68e25f371f8744d5b" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 36081 } } peers { permanent_uuid: "fbf33decd3e4423b86d9cc954f957270" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 33561 } } peers { permanent_uuid: "2b08d7c01e224bf2aaefad82a6e549db" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 39203 } }
I20260619 14:36:04.709038 20101 raft_consensus.cc:374] T bb0502fac32c47778f41ba3b6793ce85 P fbf33decd3e4423b86d9cc954f957270 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:36:04.709127 20101 raft_consensus.cc:729] T bb0502fac32c47778f41ba3b6793ce85 P fbf33decd3e4423b86d9cc954f957270 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbf33decd3e4423b86d9cc954f957270, State: Initialized, Role: FOLLOWER
I20260619 14:36:04.709165 19951 heartbeater.cc:503] Master 127.2.2.62:32935 was elected leader, sending a full tablet report...
I20260619 14:36:04.709527 20099 ts_tablet_manager.cc:1428] T bb0502fac32c47778f41ba3b6793ce85 P 2b08d7c01e224bf2aaefad82a6e549db: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260619 14:36:04.709707 20101 consensus_queue.cc:260] T bb0502fac32c47778f41ba3b6793ce85 P fbf33decd3e4423b86d9cc954f957270 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28101784cc0f42a68e25f371f8744d5b" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 36081 } } peers { permanent_uuid: "fbf33decd3e4423b86d9cc954f957270" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 33561 } } peers { permanent_uuid: "2b08d7c01e224bf2aaefad82a6e549db" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 39203 } }
I20260619 14:36:04.710412 19820 heartbeater.cc:503] Master 127.2.2.62:32935 was elected leader, sending a full tablet report...
I20260619 14:36:04.710708 20101 ts_tablet_manager.cc:1428] T bb0502fac32c47778f41ba3b6793ce85 P fbf33decd3e4423b86d9cc954f957270: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 14:36:04.711401 20100 raft_consensus.cc:348] T bb0502fac32c47778f41ba3b6793ce85 P 28101784cc0f42a68e25f371f8744d5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28101784cc0f42a68e25f371f8744d5b" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 36081 } } peers { permanent_uuid: "fbf33decd3e4423b86d9cc954f957270" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 33561 } } peers { permanent_uuid: "2b08d7c01e224bf2aaefad82a6e549db" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 39203 } }
I20260619 14:36:04.711680 20100 raft_consensus.cc:374] T bb0502fac32c47778f41ba3b6793ce85 P 28101784cc0f42a68e25f371f8744d5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:36:04.711730 20100 raft_consensus.cc:729] T bb0502fac32c47778f41ba3b6793ce85 P 28101784cc0f42a68e25f371f8744d5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28101784cc0f42a68e25f371f8744d5b, State: Initialized, Role: FOLLOWER
I20260619 14:36:04.712152 20100 consensus_queue.cc:260] T bb0502fac32c47778f41ba3b6793ce85 P 28101784cc0f42a68e25f371f8744d5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28101784cc0f42a68e25f371f8744d5b" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 36081 } } peers { permanent_uuid: "fbf33decd3e4423b86d9cc954f957270" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 33561 } } peers { permanent_uuid: "2b08d7c01e224bf2aaefad82a6e549db" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 39203 } }
I20260619 14:36:04.712779 20082 heartbeater.cc:503] Master 127.2.2.62:32935 was elected leader, sending a full tablet report...
I20260619 14:36:04.713078 20100 ts_tablet_manager.cc:1428] T bb0502fac32c47778f41ba3b6793ce85 P 28101784cc0f42a68e25f371f8744d5b: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
W20260619 14:36:04.715723 19952 tablet.cc:2369] T bb0502fac32c47778f41ba3b6793ce85 P 2b08d7c01e224bf2aaefad82a6e549db: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:36:04.764861 20106 raft_consensus.cc:482] T bb0502fac32c47778f41ba3b6793ce85 P fbf33decd3e4423b86d9cc954f957270 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:36:04.765062 20106 raft_consensus.cc:504] T bb0502fac32c47778f41ba3b6793ce85 P fbf33decd3e4423b86d9cc954f957270 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28101784cc0f42a68e25f371f8744d5b" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 36081 } } peers { permanent_uuid: "fbf33decd3e4423b86d9cc954f957270" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 33561 } } peers { permanent_uuid: "2b08d7c01e224bf2aaefad82a6e549db" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 39203 } }
W20260619 14:36:04.765437 19821 tablet.cc:2369] T bb0502fac32c47778f41ba3b6793ce85 P fbf33decd3e4423b86d9cc954f957270: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:36:04.766392 20106 leader_election.cc:288] T bb0502fac32c47778f41ba3b6793ce85 P fbf33decd3e4423b86d9cc954f957270 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28101784cc0f42a68e25f371f8744d5b (127.2.2.3:36081), 2b08d7c01e224bf2aaefad82a6e549db (127.2.2.2:39203)
I20260619 14:36:04.769913 20094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:04.766787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:47643 (local address 127.2.2.3:36081)
0619 14:36:04.766922 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:04.766928 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:04.766945 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:36:04.767008 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:04.767010 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:04.767048 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:04.767147 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:36:04.767430 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:04.767922 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:04.768969 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:04.769363 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:04.769483 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:04.769542 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:04.769719 (+ 177us) server_negotiation.cc:299] Negotiation successful
0619 14:36:04.769790 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 14:36:04.769913 19822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:04.766580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.3:36081 (local address 127.2.2.1:47643)
0619 14:36:04.766714 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:04.766731 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:36:04.766835 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:36:04.767162 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:36:04.767165 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:36:04.767184 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:36:04.767313 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:04.767319 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:04.768037 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:04.768042 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:04.768842 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:04.768850 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:04.769486 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:04.769490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:04.769589 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:04.769616 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 14:36:04.769704 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 14:36:04.769784 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 14:36:04.770100 20108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:04.766677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.2:39203 (local address 127.2.2.1:32881)
0619 14:36:04.767040 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:04.767049 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 14:36:04.767154 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:36:04.767443 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:36:04.767446 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:36:04.767461 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:36:04.767584 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:04.767589 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:04.768273 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:04.768280 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:04.769053 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:04.769062 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:04.769673 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:04.769676 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:04.769781 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:04.769806 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 14:36:04.769892 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 14:36:04.769960 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":129,"threads_started":1}
I20260619 14:36:04.770080 20097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:04.766794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:32881 (local address 127.2.2.2:39203)
0619 14:36:04.766926 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:04.766932 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:04.767127 (+ 195us) server_negotiation.cc:407] Connection header received
0619 14:36:04.767301 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:04.767303 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:04.767346 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:04.767432 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:36:04.767692 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:04.768163 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:04.769175 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:04.769556 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:04.769670 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:04.769725 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:04.769906 (+ 181us) server_negotiation.cc:299] Negotiation successful
0619 14:36:04.769972 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 14:36:04.770828 19905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb0502fac32c47778f41ba3b6793ce85" candidate_uuid: "fbf33decd3e4423b86d9cc954f957270" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b08d7c01e224bf2aaefad82a6e549db" is_pre_election: true
I20260619 14:36:04.770825 20036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb0502fac32c47778f41ba3b6793ce85" candidate_uuid: "fbf33decd3e4423b86d9cc954f957270" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28101784cc0f42a68e25f371f8744d5b" is_pre_election: true
I20260619 14:36:04.771135 19905 raft_consensus.cc:2454] T bb0502fac32c47778f41ba3b6793ce85 P 2b08d7c01e224bf2aaefad82a6e549db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbf33decd3e4423b86d9cc954f957270 in term 0.
I20260619 14:36:04.771135 20036 raft_consensus.cc:2454] T bb0502fac32c47778f41ba3b6793ce85 P 28101784cc0f42a68e25f371f8744d5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbf33decd3e4423b86d9cc954f957270 in term 0.
I20260619 14:36:04.771633 19707 leader_election.cc:302] T bb0502fac32c47778f41ba3b6793ce85 P fbf33decd3e4423b86d9cc954f957270 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28101784cc0f42a68e25f371f8744d5b, fbf33decd3e4423b86d9cc954f957270; no voters:
I20260619 14:36:04.771912 20106 raft_consensus.cc:2781] T bb0502fac32c47778f41ba3b6793ce85 P fbf33decd3e4423b86d9cc954f957270 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:36:04.772012 20106 raft_consensus.cc:482] T bb0502fac32c47778f41ba3b6793ce85 P fbf33decd3e4423b86d9cc954f957270 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:36:04.772094 20106 raft_consensus.cc:3037] T bb0502fac32c47778f41ba3b6793ce85 P fbf33decd3e4423b86d9cc954f957270 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:36:04.773116 20106 raft_consensus.cc:504] T bb0502fac32c47778f41ba3b6793ce85 P fbf33decd3e4423b86d9cc954f957270 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28101784cc0f42a68e25f371f8744d5b" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 36081 } } peers { permanent_uuid: "fbf33decd3e4423b86d9cc954f957270" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 33561 } } peers { permanent_uuid: "2b08d7c01e224bf2aaefad82a6e549db" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 39203 } }
I20260619 14:36:04.773461 20106 leader_election.cc:288] T bb0502fac32c47778f41ba3b6793ce85 P fbf33decd3e4423b86d9cc954f957270 [CANDIDATE]: Term 1 election: Requested vote from peers 28101784cc0f42a68e25f371f8744d5b (127.2.2.3:36081), 2b08d7c01e224bf2aaefad82a6e549db (127.2.2.2:39203)
I20260619 14:36:04.773823 20036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb0502fac32c47778f41ba3b6793ce85" candidate_uuid: "fbf33decd3e4423b86d9cc954f957270" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28101784cc0f42a68e25f371f8744d5b"
I20260619 14:36:04.773834 19905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb0502fac32c47778f41ba3b6793ce85" candidate_uuid: "fbf33decd3e4423b86d9cc954f957270" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b08d7c01e224bf2aaefad82a6e549db"
I20260619 14:36:04.773948 20036 raft_consensus.cc:3037] T bb0502fac32c47778f41ba3b6793ce85 P 28101784cc0f42a68e25f371f8744d5b [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:36:04.773967 19905 raft_consensus.cc:3037] T bb0502fac32c47778f41ba3b6793ce85 P 2b08d7c01e224bf2aaefad82a6e549db [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:36:04.775133 20036 raft_consensus.cc:2454] T bb0502fac32c47778f41ba3b6793ce85 P 28101784cc0f42a68e25f371f8744d5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbf33decd3e4423b86d9cc954f957270 in term 1.
I20260619 14:36:04.775178 19905 raft_consensus.cc:2454] T bb0502fac32c47778f41ba3b6793ce85 P 2b08d7c01e224bf2aaefad82a6e549db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbf33decd3e4423b86d9cc954f957270 in term 1.
I20260619 14:36:04.775466 19707 leader_election.cc:302] T bb0502fac32c47778f41ba3b6793ce85 P fbf33decd3e4423b86d9cc954f957270 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28101784cc0f42a68e25f371f8744d5b, fbf33decd3e4423b86d9cc954f957270; no voters:
I20260619 14:36:04.775650 20106 raft_consensus.cc:2781] T bb0502fac32c47778f41ba3b6793ce85 P fbf33decd3e4423b86d9cc954f957270 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:36:04.775899 20106 raft_consensus.cc:686] T bb0502fac32c47778f41ba3b6793ce85 P fbf33decd3e4423b86d9cc954f957270 [term 1 LEADER]: Becoming Leader. State: Replica: fbf33decd3e4423b86d9cc954f957270, State: Running, Role: LEADER
I20260619 14:36:04.776266 20106 consensus_queue.cc:237] T bb0502fac32c47778f41ba3b6793ce85 P fbf33decd3e4423b86d9cc954f957270 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28101784cc0f42a68e25f371f8744d5b" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 36081 } } peers { permanent_uuid: "fbf33decd3e4423b86d9cc954f957270" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 33561 } } peers { permanent_uuid: "2b08d7c01e224bf2aaefad82a6e549db" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 39203 } }
I20260619 14:36:04.779606 19631 catalog_manager.cc:5495] T bb0502fac32c47778f41ba3b6793ce85 P fbf33decd3e4423b86d9cc954f957270 reported cstate change: term changed from 0 to 1, leader changed from <none> to fbf33decd3e4423b86d9cc954f957270 (127.2.2.1). New cstate: current_term: 1 leader_uuid: "fbf33decd3e4423b86d9cc954f957270" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28101784cc0f42a68e25f371f8744d5b" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 36081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbf33decd3e4423b86d9cc954f957270" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 33561 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b08d7c01e224bf2aaefad82a6e549db" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 39203 } health_report { overall_health: UNKNOWN } } }
I20260619 14:36:04.806473 20098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:04.803274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60128 (local address 127.2.2.1:33561)
0619 14:36:04.803403 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:04.803406 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:04.803418 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:36:04.803494 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:04.803496 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:04.803551 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:04.803643 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:36:04.803844 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:04.804334 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:04.805134 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:04.805347 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:04.805424 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:04.805662 (+ 238us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:36:04.806082 (+ 420us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:36:04.806244 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:04.806296 (+ 52us) server_negotiation.cc:299] Negotiation successful
0619 14:36:04.806343 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 14:36:04.812284 19905 raft_consensus.cc:1261] T bb0502fac32c47778f41ba3b6793ce85 P 2b08d7c01e224bf2aaefad82a6e549db [term 1 FOLLOWER]: Refusing update from remote peer fbf33decd3e4423b86d9cc954f957270: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:36:04.812296 20036 raft_consensus.cc:1261] T bb0502fac32c47778f41ba3b6793ce85 P 28101784cc0f42a68e25f371f8744d5b [term 1 FOLLOWER]: Refusing update from remote peer fbf33decd3e4423b86d9cc954f957270: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:36:04.812987 20109 consensus_queue.cc:1035] T bb0502fac32c47778f41ba3b6793ce85 P fbf33decd3e4423b86d9cc954f957270 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28101784cc0f42a68e25f371f8744d5b" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 36081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:36:04.813169 20106 consensus_queue.cc:1035] T bb0502fac32c47778f41ba3b6793ce85 P fbf33decd3e4423b86d9cc954f957270 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b08d7c01e224bf2aaefad82a6e549db" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 39203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:36:04.821099 20113 mvcc.cc:205] Tried to move back new op lower bound from 7298579516664066048 to 7298579516526534656. Current Snapshot: MvccSnapshot[applied={T|T < 7298579516664066048}]
I20260619 14:36:04.821208 20114 mvcc.cc:205] Tried to move back new op lower bound from 7298579516664066048 to 7298579516526534656. Current Snapshot: MvccSnapshot[applied={T|T < 7298579516664066048}]
I20260619 14:36:04.829473 19631 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:36:04.829658 19631 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:36:04.832266 19631 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b90d0ae82e5a4bc69f05c8bfc4b47258: Sending DeleteTablet for 3 replicas of tablet bb0502fac32c47778f41ba3b6793ce85
I20260619 14:36:04.833235 19885 tablet_service.cc:1514] Processing DeleteTablet for tablet bb0502fac32c47778f41ba3b6793ce85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:36:04 UTC) from {username='slave'} at 127.0.0.1:39762
I20260619 14:36:04.833256 20016 tablet_service.cc:1514] Processing DeleteTablet for tablet bb0502fac32c47778f41ba3b6793ce85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:36:04 UTC) from {username='slave'} at 127.0.0.1:52964
I20260619 14:36:04.833890 20122 tablet_replica.cc:330] T bb0502fac32c47778f41ba3b6793ce85 P 28101784cc0f42a68e25f371f8744d5b: stopping tablet replica
I20260619 14:36:04.833999 19754 tablet_service.cc:1514] Processing DeleteTablet for tablet bb0502fac32c47778f41ba3b6793ce85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:36:04 UTC) from {username='slave'} at 127.0.0.1:60112
I20260619 14:36:04.834296 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 19692
I20260619 14:36:04.834389 20122 raft_consensus.cc:2229] T bb0502fac32c47778f41ba3b6793ce85 P 28101784cc0f42a68e25f371f8744d5b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:36:04.834717 20122 raft_consensus.cc:2258] T bb0502fac32c47778f41ba3b6793ce85 P 28101784cc0f42a68e25f371f8744d5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:36:04.836414 20122 ts_tablet_manager.cc:1905] T bb0502fac32c47778f41ba3b6793ce85 P 28101784cc0f42a68e25f371f8744d5b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:36:04.836609 20123 tablet_replica.cc:330] T bb0502fac32c47778f41ba3b6793ce85 P 2b08d7c01e224bf2aaefad82a6e549db: stopping tablet replica
I20260619 14:36:04.836918 20123 raft_consensus.cc:2229] T bb0502fac32c47778f41ba3b6793ce85 P 2b08d7c01e224bf2aaefad82a6e549db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:36:04.837222 20123 raft_consensus.cc:2258] T bb0502fac32c47778f41ba3b6793ce85 P 2b08d7c01e224bf2aaefad82a6e549db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:36:04.838581 20123 ts_tablet_manager.cc:1905] T bb0502fac32c47778f41ba3b6793ce85 P 2b08d7c01e224bf2aaefad82a6e549db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:36:04.840139 20122 ts_tablet_manager.cc:1918] T bb0502fac32c47778f41ba3b6793ce85 P 28101784cc0f42a68e25f371f8744d5b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:36:04.840256 20122 log.cc:1196] T bb0502fac32c47778f41ba3b6793ce85 P 28101784cc0f42a68e25f371f8744d5b: Deleting WAL directory at /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/ts-2/wal/wals/bb0502fac32c47778f41ba3b6793ce85
I20260619 14:36:04.840688 20122 ts_tablet_manager.cc:1939] T bb0502fac32c47778f41ba3b6793ce85 P 28101784cc0f42a68e25f371f8744d5b: Deleting consensus metadata
I20260619 14:36:04.840868 20123 ts_tablet_manager.cc:1918] T bb0502fac32c47778f41ba3b6793ce85 P 2b08d7c01e224bf2aaefad82a6e549db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:36:04.840978 20123 log.cc:1196] T bb0502fac32c47778f41ba3b6793ce85 P 2b08d7c01e224bf2aaefad82a6e549db: Deleting WAL directory at /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879514993456-2056-0/minicluster-data/ts-1/wal/wals/bb0502fac32c47778f41ba3b6793ce85
I20260619 14:36:04.841255 20123 ts_tablet_manager.cc:1939] T bb0502fac32c47778f41ba3b6793ce85 P 2b08d7c01e224bf2aaefad82a6e549db: Deleting consensus metadata
I20260619 14:36:04.842442 19618 catalog_manager.cc:4841] TS 2b08d7c01e224bf2aaefad82a6e549db (127.2.2.2:39203): tablet bb0502fac32c47778f41ba3b6793ce85 (table test-table [id=25666854fe4e4537ac89c6a71f49c8e3]) successfully deleted
W20260619 14:36:04.842538 19616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.2.1:33561 (error 108)
I20260619 14:36:04.842707 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 19823
I20260619 14:36:04.843050 19616 catalog_manager.cc:4841] TS 28101784cc0f42a68e25f371f8744d5b (127.2.2.3:36081): tablet bb0502fac32c47778f41ba3b6793ce85 (table test-table [id=25666854fe4e4537ac89c6a71f49c8e3]) successfully deleted
I20260619 14:36:04.843943 20093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:04.843667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.1:33561 (local address 127.0.0.1:60132)
0619 14:36:04.843800 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:04.843872 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.1:33561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 14:36:04.844156 19616 catalog_manager.cc:4584] TS fbf33decd3e4423b86d9cc954f957270 (127.2.2.1:33561): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bb0502fac32c47778f41ba3b6793ce85: Network error: Client connection negotiation failed: client connection to 127.2.2.1:33561: connect: Connection refused (error 111)
I20260619 14:36:04.848762 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 19954
I20260619 14:36:04.854020 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 19601
2026-06-19T14:36:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1055 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:36:04 dist-test-slave-70rm krb5kdc[20128](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:36:04 dist-test-slave-70rm krb5kdc[20128](info): set up 2 sockets
Jun 19 14:36:04 dist-test-slave-70rm krb5kdc[20128](info): commencing operation
krb5kdc: starting...
W20260619 14:36:05.890344 2056 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Jun 19 14:36:05 dist-test-slave-70rm krb5kdc[20128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879765, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:36:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:36:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.2.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab.
Entry for principal kudu/127.2.2.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab.
I20260619 14:36:06.008605 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:46665
--webserver_interface=127.2.2.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:33143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.2.62:46665
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:36:06.118111 20141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:36:06.118430 20141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:36:06.118518 20141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:36:06.122695 20141 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:36:06.122774 20141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:36:06.122795 20141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:36:06.122812 20141 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:36:06.122881 20141 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:36:06.128448 20141 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:33143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.2.62:46665
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:46665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.2.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:36:06.129566 20141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:36:06.130641 20141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:36:06.139659 20149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:06.139690 20146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:06.139683 20141 server_base.cc:1034] running on GCE node
W20260619 14:36:06.139659 20147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:06.140674 20141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:36:06.141827 20141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:36:06.143015 20141 hybrid_clock.cc:648] HybridClock initialized: now 1781879766142990 us; error 49 us; skew 500 ppm
Jun 19 14:36:06 dist-test-slave-70rm krb5kdc[20128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879766, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:36:06.145794 20141 init.cc:377] Logged in from keytab as kudu/127.2.2.62@KRBTEST.COM (short username kudu)
I20260619 14:36:06.146727 20141 webserver.cc:460] Webserver started at http://127.2.2.62:45257/ using document root <none> and password file <none>
I20260619 14:36:06.147307 20141 fs_manager.cc:359] Metadata directory not provided
I20260619 14:36:06.147403 20141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:36:06.147632 20141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:36:06.149631 20141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/master-0/data/instance:
uuid: "e59d3bea426f478f8f836c77176239a1"
format_stamp: "Formatted at 2026-06-19 14:36:06 on dist-test-slave-70rm"
server_key: "e6ea66d3a0ba98f8ea5d5887fa57cfc3"
server_key_iv: "8e7dd2ab267035a8400e3afa9037b714"
server_key_version: "encryptionkey@0"
I20260619 14:36:06.150254 20141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/master-0/wal/instance:
uuid: "e59d3bea426f478f8f836c77176239a1"
format_stamp: "Formatted at 2026-06-19 14:36:06 on dist-test-slave-70rm"
server_key: "e6ea66d3a0ba98f8ea5d5887fa57cfc3"
server_key_iv: "8e7dd2ab267035a8400e3afa9037b714"
server_key_version: "encryptionkey@0"
I20260619 14:36:06.153697 20141 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 14:36:06.156114 20156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:06.157331 20141 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:36:06.157488 20141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/master-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/master-0/wal
uuid: "e59d3bea426f478f8f836c77176239a1"
format_stamp: "Formatted at 2026-06-19 14:36:06 on dist-test-slave-70rm"
server_key: "e6ea66d3a0ba98f8ea5d5887fa57cfc3"
server_key_iv: "8e7dd2ab267035a8400e3afa9037b714"
server_key_version: "encryptionkey@0"
I20260619 14:36:06.157616 20141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:36:06.201702 20141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:36:06.205189 20141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:36:06.205407 20141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:36:06.212479 20141 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.62:46665
I20260619 14:36:06.212504 20208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.62:46665 every 8 connection(s)
I20260619 14:36:06.213452 20141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
I20260619 14:36:06.215361 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 20141
I20260619 14:36:06.215471 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/master-0/wal/instance
I20260619 14:36:06.215776 2056 external_mini_cluster.cc:1428] Setting key ccc04cf98a90b2d2c07772add07de5e9
I20260619 14:36:06.216327 20209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:36:06.222273 20209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e59d3bea426f478f8f836c77176239a1: Bootstrap starting.
Jun 19 14:36:06 dist-test-slave-70rm krb5kdc[20128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879765, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:36:06.224730 20209 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e59d3bea426f478f8f836c77176239a1: Neither blocks nor log segments found. Creating new log.
I20260619 14:36:06.225459 20209 log.cc:824] T 00000000000000000000000000000000 P e59d3bea426f478f8f836c77176239a1: Log is configured to *not* fsync() on all Append() calls
I20260619 14:36:06.227399 20209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e59d3bea426f478f8f836c77176239a1: No bootstrap required, opened a new log
I20260619 14:36:06.229839 20212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:06.217158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46008 (local address 127.2.2.62:46665)
0619 14:36:06.217606 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:06.217621 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:06.217655 (+ 34us) server_negotiation.cc:407] Connection header received
0619 14:36:06.218368 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:06.218385 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:06.218711 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:06.219043 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:06.219853 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:06.220845 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:06.221701 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:06.222258 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:06.222424 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:06.225147 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:06.225153 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:06.225167 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:36:06.225197 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:06.227005 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:06.227469 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:06.227474 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:06.227479 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:06.227631 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:06.227918 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:06.227922 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:06.227924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:06.228322 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:06.228512 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:06.228826 (+ 314us) server_negotiation.cc:299] Negotiation successful
0619 14:36:06.229121 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":145,"threads_started":1}
I20260619 14:36:06.231276 20209 raft_consensus.cc:348] T 00000000000000000000000000000000 P e59d3bea426f478f8f836c77176239a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e59d3bea426f478f8f836c77176239a1" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 46665 } }
I20260619 14:36:06.231566 20209 raft_consensus.cc:374] T 00000000000000000000000000000000 P e59d3bea426f478f8f836c77176239a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:36:06.231637 20209 raft_consensus.cc:729] T 00000000000000000000000000000000 P e59d3bea426f478f8f836c77176239a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e59d3bea426f478f8f836c77176239a1, State: Initialized, Role: FOLLOWER
I20260619 14:36:06.232132 20209 consensus_queue.cc:260] T 00000000000000000000000000000000 P e59d3bea426f478f8f836c77176239a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e59d3bea426f478f8f836c77176239a1" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 46665 } }
I20260619 14:36:06.232283 20209 raft_consensus.cc:388] T 00000000000000000000000000000000 P e59d3bea426f478f8f836c77176239a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:36:06.232334 20209 raft_consensus.cc:482] T 00000000000000000000000000000000 P e59d3bea426f478f8f836c77176239a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:36:06.232406 20209 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e59d3bea426f478f8f836c77176239a1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:36:06.233291 20209 raft_consensus.cc:504] T 00000000000000000000000000000000 P e59d3bea426f478f8f836c77176239a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e59d3bea426f478f8f836c77176239a1" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 46665 } }
I20260619 14:36:06.233570 20209 leader_election.cc:302] T 00000000000000000000000000000000 P e59d3bea426f478f8f836c77176239a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e59d3bea426f478f8f836c77176239a1; no voters:
I20260619 14:36:06.233824 20209 leader_election.cc:288] T 00000000000000000000000000000000 P e59d3bea426f478f8f836c77176239a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:36:06.233935 20214 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e59d3bea426f478f8f836c77176239a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:36:06.234205 20214 raft_consensus.cc:686] T 00000000000000000000000000000000 P e59d3bea426f478f8f836c77176239a1 [term 1 LEADER]: Becoming Leader. State: Replica: e59d3bea426f478f8f836c77176239a1, State: Running, Role: LEADER
I20260619 14:36:06.234544 20214 consensus_queue.cc:237] T 00000000000000000000000000000000 P e59d3bea426f478f8f836c77176239a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e59d3bea426f478f8f836c77176239a1" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 46665 } }
I20260619 14:36:06.235033 20209 sys_catalog.cc:564] T 00000000000000000000000000000000 P e59d3bea426f478f8f836c77176239a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:36:06.236198 20215 sys_catalog.cc:455] T 00000000000000000000000000000000 P e59d3bea426f478f8f836c77176239a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e59d3bea426f478f8f836c77176239a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e59d3bea426f478f8f836c77176239a1" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 46665 } } }
I20260619 14:36:06.236198 20216 sys_catalog.cc:455] T 00000000000000000000000000000000 P e59d3bea426f478f8f836c77176239a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e59d3bea426f478f8f836c77176239a1. Latest consensus state: current_term: 1 leader_uuid: "e59d3bea426f478f8f836c77176239a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e59d3bea426f478f8f836c77176239a1" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 46665 } } }
I20260619 14:36:06.236421 20215 sys_catalog.cc:458] T 00000000000000000000000000000000 P e59d3bea426f478f8f836c77176239a1 [sys.catalog]: This master's current role is: LEADER
I20260619 14:36:06.236510 20216 sys_catalog.cc:458] T 00000000000000000000000000000000 P e59d3bea426f478f8f836c77176239a1 [sys.catalog]: This master's current role is: LEADER
W20260619 14:36:06.239912 20229 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e59d3bea426f478f8f836c77176239a1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:36:06.239980 20229 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:36:06.240059 20225 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:36:06.240805 20225 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:36:06.245895 20225 catalog_manager.cc:1347] Generated new cluster ID: d9be44e6e08f47ccab42c18aa46a2012
I20260619 14:36:06.245963 20225 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:36:06.260279 20225 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:36:06.261619 20225 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:36:06.275228 20225 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e59d3bea426f478f8f836c77176239a1: Generated new TSK 0
I20260619 14:36:06.276120 20225 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.2.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab.
Entry for principal kudu/127.2.2.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab.
I20260619 14:36:06.309054 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.1:0
--local_ip_for_outbound_sockets=127.2.2.1
--webserver_interface=127.2.2.1
--webserver_port=0
--tserver_master_addrs=127.2.2.62:46665
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab
--principal=kudu/127.2.2.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:33143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:36:06.422781 20235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:36:06.423074 20235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:36:06.423161 20235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:36:06.427214 20235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:36:06.427299 20235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:36:06.427407 20235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.1
I20260619 14:36:06.432654 20235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:33143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab
--principal=kudu/127.2.2.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.2.1
--webserver_port=0
--tserver_master_addrs=127.2.2.62:46665
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.1
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:36:06.433696 20235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:36:06.434820 20235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:36:06.444525 20241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:06.444537 20240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:06.444598 20235 server_base.cc:1034] running on GCE node
W20260619 14:36:06.444622 20243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:06.445289 20235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:36:06.445955 20235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:36:06.447168 20235 hybrid_clock.cc:648] HybridClock initialized: now 1781879766447152 us; error 42 us; skew 500 ppm
Jun 19 14:36:06 dist-test-slave-70rm krb5kdc[20128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879766, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:36:06.449900 20235 init.cc:377] Logged in from keytab as kudu/127.2.2.1@KRBTEST.COM (short username kudu)
I20260619 14:36:06.450790 20235 webserver.cc:460] Webserver started at http://127.2.2.1:35161/ using document root <none> and password file <none>
I20260619 14:36:06.451313 20235 fs_manager.cc:359] Metadata directory not provided
I20260619 14:36:06.451370 20235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:36:06.451534 20235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:36:06.453610 20235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/ts-0/data/instance:
uuid: "ffb58cb1737b40ad84c7e4616ef7e0d2"
format_stamp: "Formatted at 2026-06-19 14:36:06 on dist-test-slave-70rm"
server_key: "219e6d69cf81a38e66cd2ef2dabd7779"
server_key_iv: "bd4d7459841cbb19d5e7df56f2105f80"
server_key_version: "encryptionkey@0"
I20260619 14:36:06.454195 20235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/ts-0/wal/instance:
uuid: "ffb58cb1737b40ad84c7e4616ef7e0d2"
format_stamp: "Formatted at 2026-06-19 14:36:06 on dist-test-slave-70rm"
server_key: "219e6d69cf81a38e66cd2ef2dabd7779"
server_key_iv: "bd4d7459841cbb19d5e7df56f2105f80"
server_key_version: "encryptionkey@0"
I20260619 14:36:06.458019 20235 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260619 14:36:06.460533 20250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:06.461846 20235 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:36:06.461946 20235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/ts-0/wal
uuid: "ffb58cb1737b40ad84c7e4616ef7e0d2"
format_stamp: "Formatted at 2026-06-19 14:36:06 on dist-test-slave-70rm"
server_key: "219e6d69cf81a38e66cd2ef2dabd7779"
server_key_iv: "bd4d7459841cbb19d5e7df56f2105f80"
server_key_version: "encryptionkey@0"
I20260619 14:36:06.462046 20235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:36:06.482575 20235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:36:06.485670 20235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:36:06.485827 20235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:36:06.486467 20235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:36:06.487250 20235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:36:06.487298 20235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:06.487336 20235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:36:06.487358 20235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:06.496407 20235 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.1:33007
I20260619 14:36:06.496425 20363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.1:33007 every 8 connection(s)
I20260619 14:36:06.497339 20235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
I20260619 14:36:06.505841 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 20235
I20260619 14:36:06.506011 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/ts-0/wal/instance
I20260619 14:36:06.506453 2056 external_mini_cluster.cc:1428] Setting key 0bb44743e5ab89a44ce704d8f0975d53
Jun 19 14:36:06 dist-test-slave-70rm krb5kdc[20128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879766, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.1@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:36:06.511744 20212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:06.499257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:39323 (local address 127.2.2.62:46665)
0619 14:36:06.499440 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:06.499444 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:06.500281 (+ 837us) server_negotiation.cc:407] Connection header received
0619 14:36:06.501208 (+ 927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:06.501211 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:06.501265 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:06.501341 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:06.502703 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:06.503229 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:06.504381 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:06.504608 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:06.504686 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:06.507600 (+ 2914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:06.507604 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:06.507607 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:36:06.507632 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:06.509221 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:06.509901 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:06.509907 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:06.509910 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:06.510015 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:06.510416 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:06.510421 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:06.510424 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:06.510681 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:06.510807 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:06.511323 (+ 516us) server_negotiation.cc:299] Negotiation successful
0619 14:36:06.511476 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 14:36:06.512467 20366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:06.499590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46665 (local address 127.2.2.1:39323)
0619 14:36:06.500087 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:06.500135 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 14:36:06.501000 (+ 865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:36:06.501554 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:36:06.501562 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:36:06.501951 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:36:06.502556 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:06.502572 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:06.503347 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:06.503349 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:06.504275 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:06.504285 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:06.504694 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:06.504746 (+ 52us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:06.504902 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:06.505576 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:36:06.505592 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:36:06.507427 (+ 1835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:36:06.509352 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:36:06.509362 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:36:06.509382 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:36:06.509788 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:36:06.510111 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:36:06.510113 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:36:06.510115 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:36:06.510320 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:36:06.510803 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:36:06.510808 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:36:06.511057 (+ 249us) client_negotiation.cc:769] Sending connection context
0619 14:36:06.511214 (+ 157us) client_negotiation.cc:240] Negotiation successful
0619 14:36:06.511496 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":125,"threads_started":1}
I20260619 14:36:06.513758 20364 heartbeater.cc:347] Connected to a master server at 127.2.2.62:46665
I20260619 14:36:06.514058 20364 heartbeater.cc:464] Registering TS with master...
I20260619 14:36:06.514735 20364 heartbeater.cc:511] Master 127.2.2.62:46665 requested a full tablet report, sending...
I20260619 14:36:06.516341 20173 ts_manager.cc:194] Registered new tserver with Master: ffb58cb1737b40ad84c7e4616ef7e0d2 (127.2.2.1:33007)
WARNING: no policy specified for kudu/127.2.2.2@KRBTEST.COM; defaulting to no policy
I20260619 14:36:06.517511 20173 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.2.1@KRBTEST.COM'} at 127.2.2.1:39323
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab.
Entry for principal kudu/127.2.2.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab.
I20260619 14:36:06.533759 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.2:0
--local_ip_for_outbound_sockets=127.2.2.2
--webserver_interface=127.2.2.2
--webserver_port=0
--tserver_master_addrs=127.2.2.62:46665
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab
--principal=kudu/127.2.2.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:33143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:36:06.644161 20369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:36:06.644448 20369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:36:06.644492 20369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:36:06.648705 20369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:36:06.648773 20369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:36:06.648850 20369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.2
I20260619 14:36:06.654320 20369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:33143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab
--principal=kudu/127.2.2.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.2.2
--webserver_port=0
--tserver_master_addrs=127.2.2.62:46665
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.2
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:36:06.655355 20369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:36:06.656358 20369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:36:06.666671 20375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:06.666792 20369 server_base.cc:1034] running on GCE node
W20260619 14:36:06.666648 20374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:06.666740 20377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:06.667374 20369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:36:06.668097 20369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:36:06.669296 20369 hybrid_clock.cc:648] HybridClock initialized: now 1781879766669280 us; error 34 us; skew 500 ppm
Jun 19 14:36:06 dist-test-slave-70rm krb5kdc[20128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879766, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:36:06.677315 20369 init.cc:377] Logged in from keytab as kudu/127.2.2.2@KRBTEST.COM (short username kudu)
I20260619 14:36:06.678360 20369 webserver.cc:460] Webserver started at http://127.2.2.2:45901/ using document root <none> and password file <none>
I20260619 14:36:06.678959 20369 fs_manager.cc:359] Metadata directory not provided
I20260619 14:36:06.679054 20369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:36:06.679303 20369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:36:06.681423 20369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/ts-1/data/instance:
uuid: "e4cd6603d92c4cb4b076d1b268c9e61f"
format_stamp: "Formatted at 2026-06-19 14:36:06 on dist-test-slave-70rm"
server_key: "0db6427f28a9d4f8e63d2fbe780624cf"
server_key_iv: "dcc6e6770d5e5c202b84a35a3a3b305d"
server_key_version: "encryptionkey@0"
I20260619 14:36:06.682008 20369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/ts-1/wal/instance:
uuid: "e4cd6603d92c4cb4b076d1b268c9e61f"
format_stamp: "Formatted at 2026-06-19 14:36:06 on dist-test-slave-70rm"
server_key: "0db6427f28a9d4f8e63d2fbe780624cf"
server_key_iv: "dcc6e6770d5e5c202b84a35a3a3b305d"
server_key_version: "encryptionkey@0"
I20260619 14:36:06.685962 20369 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260619 14:36:06.688550 20384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:06.689787 20369 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:36:06.689924 20369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/ts-1/wal
uuid: "e4cd6603d92c4cb4b076d1b268c9e61f"
format_stamp: "Formatted at 2026-06-19 14:36:06 on dist-test-slave-70rm"
server_key: "0db6427f28a9d4f8e63d2fbe780624cf"
server_key_iv: "dcc6e6770d5e5c202b84a35a3a3b305d"
server_key_version: "encryptionkey@0"
I20260619 14:36:06.690039 20369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:36:06.707794 20369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:36:06.710783 20369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:36:06.710989 20369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:36:06.711567 20369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:36:06.712386 20369 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:36:06.712452 20369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:06.712517 20369 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:36:06.712563 20369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:06.721657 20369 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.2:33311
I20260619 14:36:06.721676 20497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.2:33311 every 8 connection(s)
I20260619 14:36:06.722640 20369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
I20260619 14:36:06.730417 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 20369
I20260619 14:36:06.730546 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/ts-1/wal/instance
I20260619 14:36:06.730865 2056 external_mini_cluster.cc:1428] Setting key 279c68550283fed2cc170594522c0ee5
Jun 19 14:36:06 dist-test-slave-70rm krb5kdc[20128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879766, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.2@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:36:06.736697 20212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:06.724489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:45149 (local address 127.2.2.62:46665)
0619 14:36:06.724632 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:06.724636 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:06.725491 (+ 855us) server_negotiation.cc:407] Connection header received
0619 14:36:06.726452 (+ 961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:06.726456 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:06.726512 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:06.726592 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:06.727730 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:06.728266 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:06.729404 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:06.729636 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:06.729751 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:06.732702 (+ 2951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:06.732713 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:06.732716 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:36:06.732740 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:06.734340 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:06.734962 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:06.734968 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:06.734972 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:06.735064 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:06.735418 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:06.735424 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:06.735428 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:06.735704 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:06.735808 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:06.736416 (+ 608us) server_negotiation.cc:299] Negotiation successful
0619 14:36:06.736539 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 14:36:06.737833 20500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:06.724882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46665 (local address 127.2.2.2:45149)
0619 14:36:06.725335 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:06.725380 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 14:36:06.726225 (+ 845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:36:06.726756 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:36:06.726764 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:36:06.727120 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:36:06.727590 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:06.727602 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:06.728396 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:06.728404 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:06.729282 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:06.729291 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:06.729735 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:06.729798 (+ 63us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:06.729964 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:06.730556 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:36:06.730580 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:36:06.732519 (+ 1939us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:36:06.734453 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:36:06.734460 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:36:06.734479 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:36:06.734861 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:36:06.735191 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:36:06.735193 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:36:06.735194 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:36:06.735330 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:36:06.735820 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:36:06.735826 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:36:06.736104 (+ 278us) client_negotiation.cc:769] Sending connection context
0619 14:36:06.736370 (+ 266us) client_negotiation.cc:240] Negotiation successful
0619 14:36:06.736703 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":115,"threads_started":1}
I20260619 14:36:06.739069 20498 heartbeater.cc:347] Connected to a master server at 127.2.2.62:46665
I20260619 14:36:06.739336 20498 heartbeater.cc:464] Registering TS with master...
I20260619 14:36:06.739831 20498 heartbeater.cc:511] Master 127.2.2.62:46665 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.2.3@KRBTEST.COM; defaulting to no policy
I20260619 14:36:06.741042 20173 ts_manager.cc:194] Registered new tserver with Master: e4cd6603d92c4cb4b076d1b268c9e61f (127.2.2.2:33311)
I20260619 14:36:06.741820 20173 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.2.2@KRBTEST.COM'} at 127.2.2.2:45149
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab.
Entry for principal kudu/127.2.2.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab.
I20260619 14:36:06.762869 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.3:0
--local_ip_for_outbound_sockets=127.2.2.3
--webserver_interface=127.2.2.3
--webserver_port=0
--tserver_master_addrs=127.2.2.62:46665
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab
--principal=kudu/127.2.2.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:33143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:36:06.872848 20503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:36:06.873148 20503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:36:06.873246 20503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:36:06.877452 20503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:36:06.877543 20503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:36:06.877627 20503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.3
I20260619 14:36:06.883143 20503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:33143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab
--principal=kudu/127.2.2.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.2.3
--webserver_port=0
--tserver_master_addrs=127.2.2.62:46665
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.3
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:36:06.884253 20503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:36:06.885310 20503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:36:06.894820 20508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:06.894953 20511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:06.894814 20509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:06.895252 20503 server_base.cc:1034] running on GCE node
I20260619 14:36:06.895699 20503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:36:06.896334 20503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:36:06.897492 20503 hybrid_clock.cc:648] HybridClock initialized: now 1781879766897459 us; error 46 us; skew 500 ppm
Jun 19 14:36:06 dist-test-slave-70rm krb5kdc[20128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879766, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:36:06.900388 20503 init.cc:377] Logged in from keytab as kudu/127.2.2.3@KRBTEST.COM (short username kudu)
I20260619 14:36:06.901301 20503 webserver.cc:460] Webserver started at http://127.2.2.3:38131/ using document root <none> and password file <none>
I20260619 14:36:06.901888 20503 fs_manager.cc:359] Metadata directory not provided
I20260619 14:36:06.901952 20503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:36:06.902271 20503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:36:06.904311 20503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/ts-2/data/instance:
uuid: "f6ea195ca6c14b0db819a2d864e7e728"
format_stamp: "Formatted at 2026-06-19 14:36:06 on dist-test-slave-70rm"
server_key: "af21950d12137209c449e8f3eab5d1bf"
server_key_iv: "ba52a71cba9eda5421f1bcaf0ff844f1"
server_key_version: "encryptionkey@0"
I20260619 14:36:06.904875 20503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/ts-2/wal/instance:
uuid: "f6ea195ca6c14b0db819a2d864e7e728"
format_stamp: "Formatted at 2026-06-19 14:36:06 on dist-test-slave-70rm"
server_key: "af21950d12137209c449e8f3eab5d1bf"
server_key_iv: "ba52a71cba9eda5421f1bcaf0ff844f1"
server_key_version: "encryptionkey@0"
I20260619 14:36:06.908674 20503 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260619 14:36:06.911149 20518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:06.912384 20503 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:36:06.912510 20503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/ts-2/wal
uuid: "f6ea195ca6c14b0db819a2d864e7e728"
format_stamp: "Formatted at 2026-06-19 14:36:06 on dist-test-slave-70rm"
server_key: "af21950d12137209c449e8f3eab5d1bf"
server_key_iv: "ba52a71cba9eda5421f1bcaf0ff844f1"
server_key_version: "encryptionkey@0"
I20260619 14:36:06.912624 20503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:36:06.925745 20503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:36:06.929229 20503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:36:06.929445 20503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:36:06.930071 20503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:36:06.930933 20503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:36:06.930979 20503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:06.931046 20503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:36:06.931090 20503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:06.940567 20503 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.3:39991
I20260619 14:36:06.940623 20631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.3:39991 every 8 connection(s)
I20260619 14:36:06.941466 20503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
I20260619 14:36:06.949170 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 20503
I20260619 14:36:06.949335 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/ts-2/wal/instance
I20260619 14:36:06.949743 2056 external_mini_cluster.cc:1428] Setting key 850bbf2738395823ee63c2d9c09ffb95
Jun 19 14:36:06 dist-test-slave-70rm krb5kdc[20128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879766, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.3@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:36:06.955575 20212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:06.943359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.3:54987 (local address 127.2.2.62:46665)
0619 14:36:06.943535 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:06.943539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:06.944406 (+ 867us) server_negotiation.cc:407] Connection header received
0619 14:36:06.945288 (+ 882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:06.945291 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:06.945338 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:06.945415 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:06.946739 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:06.947263 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:06.948443 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:06.948655 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:06.948736 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:06.952068 (+ 3332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:06.952072 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:06.952074 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:36:06.952100 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:06.953503 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:06.954033 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:06.954036 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:06.954038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:06.954114 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:06.954541 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:06.954544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:06.954545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:06.954725 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:06.954803 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:06.955340 (+ 537us) server_negotiation.cc:299] Negotiation successful
0619 14:36:06.955455 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 14:36:06.956547 20634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:06.943743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46665 (local address 127.2.2.3:54987)
0619 14:36:06.944232 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:06.944277 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 14:36:06.945072 (+ 795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:36:06.945595 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:36:06.945602 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:36:06.945977 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:36:06.946597 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:06.946612 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:06.947384 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:06.947387 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:06.948324 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:06.948334 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:06.948771 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:06.948873 (+ 102us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:06.949166 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:06.949863 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:36:06.949887 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:36:06.951880 (+ 1993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:36:06.953618 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:36:06.953623 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:36:06.953637 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:36:06.953941 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:36:06.954265 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:36:06.954268 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:36:06.954269 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:36:06.954453 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:36:06.954820 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:36:06.954826 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:36:06.955115 (+ 289us) client_negotiation.cc:769] Sending connection context
0619 14:36:06.955300 (+ 185us) client_negotiation.cc:240] Negotiation successful
0619 14:36:06.955609 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":114,"threads_started":1}
I20260619 14:36:06.957892 20632 heartbeater.cc:347] Connected to a master server at 127.2.2.62:46665
I20260619 14:36:06.958276 20632 heartbeater.cc:464] Registering TS with master...
I20260619 14:36:06.958855 20632 heartbeater.cc:511] Master 127.2.2.62:46665 requested a full tablet report, sending...
I20260619 14:36:06.959990 20173 ts_manager.cc:194] Registered new tserver with Master: f6ea195ca6c14b0db819a2d864e7e728 (127.2.2.3:39991)
I20260619 14:36:06.960577 20173 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.2.3@KRBTEST.COM'} at 127.2.2.3:54987
I20260619 14:36:06.963896 2056 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:36:06.973088 20212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:06.965799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36774 (local address 127.2.2.62:46665)
0619 14:36:06.965944 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:06.965949 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:06.966169 (+ 220us) server_negotiation.cc:407] Connection header received
0619 14:36:06.966330 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:06.966333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:06.966404 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:06.966488 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:06.967136 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:06.967631 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:06.968392 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:06.968595 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:06.968665 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:06.970074 (+ 1409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:06.970077 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:06.970079 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:36:06.970101 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:06.971422 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:06.971927 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:06.971929 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:06.971931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:06.971998 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:06.972281 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:06.972283 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:06.972284 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:06.972465 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:06.972572 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:06.972822 (+ 250us) server_negotiation.cc:299] Negotiation successful
0619 14:36:06.972896 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 14:36:06.975886 20173 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36774:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:36:06.978899 20173 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:36:06.993587 20644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:06.987088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.1:33007 (local address 127.0.0.1:54138)
0619 14:36:06.987670 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:06.987722 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 14:36:06.988379 (+ 657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:36:06.988873 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:36:06.988878 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:36:06.989005 (+ 127us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:36:06.989169 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:06.989178 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:06.990825 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:06.990840 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:06.991886 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:06.991896 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:06.993050 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:06.993054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:06.993183 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:06.993218 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 14:36:06.993299 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 14:36:06.993401 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"mutex_wait_us":20,"thread_start_us":76,"threads_started":1}
I20260619 14:36:06.993671 20645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:06.987225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54138 (local address 127.2.2.1:33007)
0619 14:36:06.988197 (+ 972us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:06.988210 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:06.988252 (+ 42us) server_negotiation.cc:407] Connection header received
0619 14:36:06.988511 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:06.988516 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:06.988702 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:06.989358 (+ 656us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:36:06.989546 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:06.990667 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:06.992027 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:06.992891 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:06.993161 (+ 270us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:06.993296 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:06.993350 (+ 54us) server_negotiation.cc:299] Negotiation successful
0619 14:36:06.993468 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":871,"thread_start_us":80,"threads_started":1}
I20260619 14:36:06.993856 20646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:06.987558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37980 (local address 127.2.2.3:39991)
0619 14:36:06.987965 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:06.987977 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:06.988001 (+ 24us) server_negotiation.cc:407] Connection header received
0619 14:36:06.988249 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:06.988254 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:06.988518 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:06.988673 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:36:06.989055 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:06.990319 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:06.992096 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:06.992891 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:06.993023 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:06.993190 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:06.993516 (+ 326us) server_negotiation.cc:299] Negotiation successful
0619 14:36:06.993679 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":133,"threads_started":1}
I20260619 14:36:06.993784 20643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:06.987170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.3:39991 (local address 127.0.0.1:37980)
0619 14:36:06.987914 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:06.987927 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:36:06.988049 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:36:06.988688 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:36:06.988694 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:36:06.988717 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:36:06.988880 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:06.988886 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:06.990461 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:06.990464 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:06.991975 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:06.991986 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:06.993254 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:06.993258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:06.993350 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:06.993378 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 14:36:06.993466 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 14:36:06.993541 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"mutex_wait_us":86,"thread_start_us":107,"threads_started":1}
I20260619 14:36:06.994395 20647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:06.987806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.2:33311 (local address 127.0.0.1:47244)
0619 14:36:06.988780 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:06.988804 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 14:36:06.988887 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:36:06.989443 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:36:06.989446 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:36:06.989464 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:36:06.989570 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:06.989593 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:06.990819 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:06.990823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:06.992146 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:06.992153 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:06.992956 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:06.992959 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:06.994003 (+ 1044us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:06.994042 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 14:36:06.994152 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 14:36:06.994230 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":907,"mutex_wait_us":164,"thread_start_us":84,"threads_started":1}
I20260619 14:36:06.994836 20648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:06.988192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47244 (local address 127.2.2.2:33311)
0619 14:36:06.988642 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:06.988654 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:06.988984 (+ 330us) server_negotiation.cc:407] Connection header received
0619 14:36:06.989122 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:06.989127 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:06.989280 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:06.989468 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:36:06.989708 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:06.990658 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:06.992257 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:06.992828 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:06.992956 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:06.993115 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:06.994610 (+ 1495us) server_negotiation.cc:299] Negotiation successful
0619 14:36:06.994714 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":144,"threads_started":1}
I20260619 14:36:06.996997 20298 tablet_service.cc:1467] Processing CreateTablet for tablet 24aa9ae983b548539ac43760f59a5cc8 (DEFAULT_TABLE table=test-table [id=7c85d3ea190843f49bda1a93586c7043]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:36:06.996999 20566 tablet_service.cc:1467] Processing CreateTablet for tablet 24aa9ae983b548539ac43760f59a5cc8 (DEFAULT_TABLE table=test-table [id=7c85d3ea190843f49bda1a93586c7043]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:36:06.997982 20298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24aa9ae983b548539ac43760f59a5cc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:36:06.997982 20566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24aa9ae983b548539ac43760f59a5cc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:36:07.001266 20432 tablet_service.cc:1467] Processing CreateTablet for tablet 24aa9ae983b548539ac43760f59a5cc8 (DEFAULT_TABLE table=test-table [id=7c85d3ea190843f49bda1a93586c7043]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:36:07.002456 20432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24aa9ae983b548539ac43760f59a5cc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:36:07.005345 20650 tablet_bootstrap.cc:492] T 24aa9ae983b548539ac43760f59a5cc8 P ffb58cb1737b40ad84c7e4616ef7e0d2: Bootstrap starting.
I20260619 14:36:07.005345 20649 tablet_bootstrap.cc:492] T 24aa9ae983b548539ac43760f59a5cc8 P f6ea195ca6c14b0db819a2d864e7e728: Bootstrap starting.
I20260619 14:36:07.007819 20650 tablet_bootstrap.cc:654] T 24aa9ae983b548539ac43760f59a5cc8 P ffb58cb1737b40ad84c7e4616ef7e0d2: Neither blocks nor log segments found. Creating new log.
I20260619 14:36:07.007819 20649 tablet_bootstrap.cc:654] T 24aa9ae983b548539ac43760f59a5cc8 P f6ea195ca6c14b0db819a2d864e7e728: Neither blocks nor log segments found. Creating new log.
I20260619 14:36:07.009088 20651 tablet_bootstrap.cc:492] T 24aa9ae983b548539ac43760f59a5cc8 P e4cd6603d92c4cb4b076d1b268c9e61f: Bootstrap starting.
I20260619 14:36:07.009195 20650 log.cc:824] T 24aa9ae983b548539ac43760f59a5cc8 P ffb58cb1737b40ad84c7e4616ef7e0d2: Log is configured to *not* fsync() on all Append() calls
I20260619 14:36:07.009195 20649 log.cc:824] T 24aa9ae983b548539ac43760f59a5cc8 P f6ea195ca6c14b0db819a2d864e7e728: Log is configured to *not* fsync() on all Append() calls
I20260619 14:36:07.011427 20651 tablet_bootstrap.cc:654] T 24aa9ae983b548539ac43760f59a5cc8 P e4cd6603d92c4cb4b076d1b268c9e61f: Neither blocks nor log segments found. Creating new log.
I20260619 14:36:07.011556 20649 tablet_bootstrap.cc:492] T 24aa9ae983b548539ac43760f59a5cc8 P f6ea195ca6c14b0db819a2d864e7e728: No bootstrap required, opened a new log
I20260619 14:36:07.011691 20650 tablet_bootstrap.cc:492] T 24aa9ae983b548539ac43760f59a5cc8 P ffb58cb1737b40ad84c7e4616ef7e0d2: No bootstrap required, opened a new log
I20260619 14:36:07.011771 20649 ts_tablet_manager.cc:1397] T 24aa9ae983b548539ac43760f59a5cc8 P f6ea195ca6c14b0db819a2d864e7e728: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 14:36:07.011853 20650 ts_tablet_manager.cc:1397] T 24aa9ae983b548539ac43760f59a5cc8 P ffb58cb1737b40ad84c7e4616ef7e0d2: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260619 14:36:07.012534 20651 log.cc:824] T 24aa9ae983b548539ac43760f59a5cc8 P e4cd6603d92c4cb4b076d1b268c9e61f: Log is configured to *not* fsync() on all Append() calls
I20260619 14:36:07.014092 20651 tablet_bootstrap.cc:492] T 24aa9ae983b548539ac43760f59a5cc8 P e4cd6603d92c4cb4b076d1b268c9e61f: No bootstrap required, opened a new log
I20260619 14:36:07.014320 20651 ts_tablet_manager.cc:1397] T 24aa9ae983b548539ac43760f59a5cc8 P e4cd6603d92c4cb4b076d1b268c9e61f: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.004s
I20260619 14:36:07.015847 20650 raft_consensus.cc:348] T 24aa9ae983b548539ac43760f59a5cc8 P ffb58cb1737b40ad84c7e4616ef7e0d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffb58cb1737b40ad84c7e4616ef7e0d2" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 33007 } } peers { permanent_uuid: "f6ea195ca6c14b0db819a2d864e7e728" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 39991 } } peers { permanent_uuid: "e4cd6603d92c4cb4b076d1b268c9e61f" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 33311 } }
I20260619 14:36:07.015847 20649 raft_consensus.cc:348] T 24aa9ae983b548539ac43760f59a5cc8 P f6ea195ca6c14b0db819a2d864e7e728 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffb58cb1737b40ad84c7e4616ef7e0d2" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 33007 } } peers { permanent_uuid: "f6ea195ca6c14b0db819a2d864e7e728" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 39991 } } peers { permanent_uuid: "e4cd6603d92c4cb4b076d1b268c9e61f" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 33311 } }
I20260619 14:36:07.016147 20650 raft_consensus.cc:374] T 24aa9ae983b548539ac43760f59a5cc8 P ffb58cb1737b40ad84c7e4616ef7e0d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:36:07.016211 20650 raft_consensus.cc:729] T 24aa9ae983b548539ac43760f59a5cc8 P ffb58cb1737b40ad84c7e4616ef7e0d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffb58cb1737b40ad84c7e4616ef7e0d2, State: Initialized, Role: FOLLOWER
I20260619 14:36:07.016147 20649 raft_consensus.cc:374] T 24aa9ae983b548539ac43760f59a5cc8 P f6ea195ca6c14b0db819a2d864e7e728 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:36:07.016271 20649 raft_consensus.cc:729] T 24aa9ae983b548539ac43760f59a5cc8 P f6ea195ca6c14b0db819a2d864e7e728 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6ea195ca6c14b0db819a2d864e7e728, State: Initialized, Role: FOLLOWER
I20260619 14:36:07.016739 20649 consensus_queue.cc:260] T 24aa9ae983b548539ac43760f59a5cc8 P f6ea195ca6c14b0db819a2d864e7e728 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffb58cb1737b40ad84c7e4616ef7e0d2" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 33007 } } peers { permanent_uuid: "f6ea195ca6c14b0db819a2d864e7e728" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 39991 } } peers { permanent_uuid: "e4cd6603d92c4cb4b076d1b268c9e61f" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 33311 } }
I20260619 14:36:07.016739 20650 consensus_queue.cc:260] T 24aa9ae983b548539ac43760f59a5cc8 P ffb58cb1737b40ad84c7e4616ef7e0d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffb58cb1737b40ad84c7e4616ef7e0d2" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 33007 } } peers { permanent_uuid: "f6ea195ca6c14b0db819a2d864e7e728" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 39991 } } peers { permanent_uuid: "e4cd6603d92c4cb4b076d1b268c9e61f" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 33311 } }
I20260619 14:36:07.017485 20651 raft_consensus.cc:348] T 24aa9ae983b548539ac43760f59a5cc8 P e4cd6603d92c4cb4b076d1b268c9e61f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffb58cb1737b40ad84c7e4616ef7e0d2" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 33007 } } peers { permanent_uuid: "f6ea195ca6c14b0db819a2d864e7e728" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 39991 } } peers { permanent_uuid: "e4cd6603d92c4cb4b076d1b268c9e61f" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 33311 } }
I20260619 14:36:07.017795 20650 ts_tablet_manager.cc:1428] T 24aa9ae983b548539ac43760f59a5cc8 P ffb58cb1737b40ad84c7e4616ef7e0d2: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 14:36:07.017795 20649 ts_tablet_manager.cc:1428] T 24aa9ae983b548539ac43760f59a5cc8 P f6ea195ca6c14b0db819a2d864e7e728: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 14:36:07.017802 20651 raft_consensus.cc:374] T 24aa9ae983b548539ac43760f59a5cc8 P e4cd6603d92c4cb4b076d1b268c9e61f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:36:07.017871 20651 raft_consensus.cc:729] T 24aa9ae983b548539ac43760f59a5cc8 P e4cd6603d92c4cb4b076d1b268c9e61f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cd6603d92c4cb4b076d1b268c9e61f, State: Initialized, Role: FOLLOWER
I20260619 14:36:07.018215 20364 heartbeater.cc:503] Master 127.2.2.62:46665 was elected leader, sending a full tablet report...
I20260619 14:36:07.018431 20651 consensus_queue.cc:260] T 24aa9ae983b548539ac43760f59a5cc8 P e4cd6603d92c4cb4b076d1b268c9e61f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffb58cb1737b40ad84c7e4616ef7e0d2" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 33007 } } peers { permanent_uuid: "f6ea195ca6c14b0db819a2d864e7e728" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 39991 } } peers { permanent_uuid: "e4cd6603d92c4cb4b076d1b268c9e61f" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 33311 } }
I20260619 14:36:07.019111 20498 heartbeater.cc:503] Master 127.2.2.62:46665 was elected leader, sending a full tablet report...
I20260619 14:36:07.019299 20632 heartbeater.cc:503] Master 127.2.2.62:46665 was elected leader, sending a full tablet report...
I20260619 14:36:07.019467 20651 ts_tablet_manager.cc:1428] T 24aa9ae983b548539ac43760f59a5cc8 P e4cd6603d92c4cb4b076d1b268c9e61f: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:36:07.020298 20655 raft_consensus.cc:482] T 24aa9ae983b548539ac43760f59a5cc8 P ffb58cb1737b40ad84c7e4616ef7e0d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:36:07.021313 20655 raft_consensus.cc:504] T 24aa9ae983b548539ac43760f59a5cc8 P ffb58cb1737b40ad84c7e4616ef7e0d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffb58cb1737b40ad84c7e4616ef7e0d2" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 33007 } } peers { permanent_uuid: "f6ea195ca6c14b0db819a2d864e7e728" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 39991 } } peers { permanent_uuid: "e4cd6603d92c4cb4b076d1b268c9e61f" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 33311 } }
I20260619 14:36:07.025215 20655 leader_election.cc:288] T 24aa9ae983b548539ac43760f59a5cc8 P ffb58cb1737b40ad84c7e4616ef7e0d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6ea195ca6c14b0db819a2d864e7e728 (127.2.2.3:39991), e4cd6603d92c4cb4b076d1b268c9e61f (127.2.2.2:33311)
I20260619 14:36:07.027827 20646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:07.023396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:36283 (local address 127.2.2.3:39991)
0619 14:36:07.023561 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:07.023565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:07.023578 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:36:07.023619 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:07.023622 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:07.023658 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:07.023732 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:36:07.024573 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:07.025334 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:07.026857 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:07.027281 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:07.027401 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:07.027462 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:07.027636 (+ 174us) server_negotiation.cc:299] Negotiation successful
0619 14:36:07.027707 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260619 14:36:07.027845 20658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:07.022742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.3:39991 (local address 127.2.2.1:36283)
0619 14:36:07.023044 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:07.023060 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:36:07.023158 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:36:07.024252 (+ 1094us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:36:07.024257 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:36:07.024278 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:36:07.024436 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:07.024444 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:07.025472 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:07.025475 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:07.026745 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:07.026753 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:07.027410 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:07.027414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:07.027504 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:07.027527 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 14:36:07.027621 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 14:36:07.027692 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":118,"threads_started":1}
I20260619 14:36:07.028309 20648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:07.024069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:59875 (local address 127.2.2.2:33311)
0619 14:36:07.024172 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:07.024176 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:07.024291 (+ 115us) server_negotiation.cc:407] Connection header received
0619 14:36:07.024493 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:07.024497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:07.024547 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:07.024603 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:36:07.025092 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:07.025779 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:07.027127 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:07.027559 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:07.027700 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:07.027766 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:07.028098 (+ 332us) server_negotiation.cc:299] Negotiation successful
0619 14:36:07.028182 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 14:36:07.028333 20659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:07.023769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.2:33311 (local address 127.2.2.1:59875)
0619 14:36:07.024198 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:07.024214 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:36:07.024363 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:36:07.024769 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:36:07.024772 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:36:07.024790 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:36:07.024930 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:07.024938 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:07.025903 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:07.025907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:07.026998 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:07.027008 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:07.027701 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:07.027705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:07.027809 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:07.027835 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 14:36:07.028121 (+ 286us) client_negotiation.cc:240] Negotiation successful
0619 14:36:07.028192 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":114,"threads_started":1}
I20260619 14:36:07.028353 20586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24aa9ae983b548539ac43760f59a5cc8" candidate_uuid: "ffb58cb1737b40ad84c7e4616ef7e0d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6ea195ca6c14b0db819a2d864e7e728" is_pre_election: true
I20260619 14:36:07.028657 20586 raft_consensus.cc:2454] T 24aa9ae983b548539ac43760f59a5cc8 P f6ea195ca6c14b0db819a2d864e7e728 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffb58cb1737b40ad84c7e4616ef7e0d2 in term 0.
I20260619 14:36:07.028967 20452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24aa9ae983b548539ac43760f59a5cc8" candidate_uuid: "ffb58cb1737b40ad84c7e4616ef7e0d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cd6603d92c4cb4b076d1b268c9e61f" is_pre_election: true
I20260619 14:36:07.029162 20254 leader_election.cc:302] T 24aa9ae983b548539ac43760f59a5cc8 P ffb58cb1737b40ad84c7e4616ef7e0d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f6ea195ca6c14b0db819a2d864e7e728, ffb58cb1737b40ad84c7e4616ef7e0d2; no voters:
I20260619 14:36:07.029228 20452 raft_consensus.cc:2454] T 24aa9ae983b548539ac43760f59a5cc8 P e4cd6603d92c4cb4b076d1b268c9e61f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffb58cb1737b40ad84c7e4616ef7e0d2 in term 0.
I20260619 14:36:07.029469 20655 raft_consensus.cc:2781] T 24aa9ae983b548539ac43760f59a5cc8 P ffb58cb1737b40ad84c7e4616ef7e0d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:36:07.029590 20655 raft_consensus.cc:482] T 24aa9ae983b548539ac43760f59a5cc8 P ffb58cb1737b40ad84c7e4616ef7e0d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:36:07.029659 20655 raft_consensus.cc:3037] T 24aa9ae983b548539ac43760f59a5cc8 P ffb58cb1737b40ad84c7e4616ef7e0d2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:36:07.030860 20655 raft_consensus.cc:504] T 24aa9ae983b548539ac43760f59a5cc8 P ffb58cb1737b40ad84c7e4616ef7e0d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffb58cb1737b40ad84c7e4616ef7e0d2" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 33007 } } peers { permanent_uuid: "f6ea195ca6c14b0db819a2d864e7e728" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 39991 } } peers { permanent_uuid: "e4cd6603d92c4cb4b076d1b268c9e61f" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 33311 } }
I20260619 14:36:07.031244 20655 leader_election.cc:288] T 24aa9ae983b548539ac43760f59a5cc8 P ffb58cb1737b40ad84c7e4616ef7e0d2 [CANDIDATE]: Term 1 election: Requested vote from peers f6ea195ca6c14b0db819a2d864e7e728 (127.2.2.3:39991), e4cd6603d92c4cb4b076d1b268c9e61f (127.2.2.2:33311)
I20260619 14:36:07.031646 20452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24aa9ae983b548539ac43760f59a5cc8" candidate_uuid: "ffb58cb1737b40ad84c7e4616ef7e0d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cd6603d92c4cb4b076d1b268c9e61f"
I20260619 14:36:07.031646 20586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24aa9ae983b548539ac43760f59a5cc8" candidate_uuid: "ffb58cb1737b40ad84c7e4616ef7e0d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6ea195ca6c14b0db819a2d864e7e728"
I20260619 14:36:07.031780 20452 raft_consensus.cc:3037] T 24aa9ae983b548539ac43760f59a5cc8 P e4cd6603d92c4cb4b076d1b268c9e61f [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:36:07.031780 20586 raft_consensus.cc:3037] T 24aa9ae983b548539ac43760f59a5cc8 P f6ea195ca6c14b0db819a2d864e7e728 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:36:07.033085 20586 raft_consensus.cc:2454] T 24aa9ae983b548539ac43760f59a5cc8 P f6ea195ca6c14b0db819a2d864e7e728 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffb58cb1737b40ad84c7e4616ef7e0d2 in term 1.
I20260619 14:36:07.033085 20452 raft_consensus.cc:2454] T 24aa9ae983b548539ac43760f59a5cc8 P e4cd6603d92c4cb4b076d1b268c9e61f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffb58cb1737b40ad84c7e4616ef7e0d2 in term 1.
I20260619 14:36:07.033385 20254 leader_election.cc:302] T 24aa9ae983b548539ac43760f59a5cc8 P ffb58cb1737b40ad84c7e4616ef7e0d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f6ea195ca6c14b0db819a2d864e7e728, ffb58cb1737b40ad84c7e4616ef7e0d2; no voters:
I20260619 14:36:07.033607 20655 raft_consensus.cc:2781] T 24aa9ae983b548539ac43760f59a5cc8 P ffb58cb1737b40ad84c7e4616ef7e0d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:36:07.033802 20655 raft_consensus.cc:686] T 24aa9ae983b548539ac43760f59a5cc8 P ffb58cb1737b40ad84c7e4616ef7e0d2 [term 1 LEADER]: Becoming Leader. State: Replica: ffb58cb1737b40ad84c7e4616ef7e0d2, State: Running, Role: LEADER
I20260619 14:36:07.034175 20655 consensus_queue.cc:237] T 24aa9ae983b548539ac43760f59a5cc8 P ffb58cb1737b40ad84c7e4616ef7e0d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffb58cb1737b40ad84c7e4616ef7e0d2" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 33007 } } peers { permanent_uuid: "f6ea195ca6c14b0db819a2d864e7e728" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 39991 } } peers { permanent_uuid: "e4cd6603d92c4cb4b076d1b268c9e61f" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 33311 } }
I20260619 14:36:07.037601 20171 catalog_manager.cc:5495] T 24aa9ae983b548539ac43760f59a5cc8 P ffb58cb1737b40ad84c7e4616ef7e0d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to ffb58cb1737b40ad84c7e4616ef7e0d2 (127.2.2.1). New cstate: current_term: 1 leader_uuid: "ffb58cb1737b40ad84c7e4616ef7e0d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffb58cb1737b40ad84c7e4616ef7e0d2" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 33007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6ea195ca6c14b0db819a2d864e7e728" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 39991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4cd6603d92c4cb4b076d1b268c9e61f" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 33311 } health_report { overall_health: UNKNOWN } } }
I20260619 14:36:07.058600 20645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:07.054888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54144 (local address 127.2.2.1:33007)
0619 14:36:07.055043 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:07.055050 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:07.055069 (+ 19us) server_negotiation.cc:407] Connection header received
0619 14:36:07.055160 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:07.055166 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:07.055268 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:07.055382 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:36:07.055702 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:07.056211 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:07.057082 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:07.057309 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:07.057422 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:07.057737 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:36:07.058160 (+ 423us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:36:07.058331 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:07.058416 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 14:36:07.058454 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 14:36:07.064780 20452 raft_consensus.cc:1261] T 24aa9ae983b548539ac43760f59a5cc8 P e4cd6603d92c4cb4b076d1b268c9e61f [term 1 FOLLOWER]: Refusing update from remote peer ffb58cb1737b40ad84c7e4616ef7e0d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:36:07.064757 20586 raft_consensus.cc:1261] T 24aa9ae983b548539ac43760f59a5cc8 P f6ea195ca6c14b0db819a2d864e7e728 [term 1 FOLLOWER]: Refusing update from remote peer ffb58cb1737b40ad84c7e4616ef7e0d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:36:07.065584 20655 consensus_queue.cc:1035] T 24aa9ae983b548539ac43760f59a5cc8 P ffb58cb1737b40ad84c7e4616ef7e0d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6ea195ca6c14b0db819a2d864e7e728" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 39991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:36:07.065788 20660 consensus_queue.cc:1035] T 24aa9ae983b548539ac43760f59a5cc8 P ffb58cb1737b40ad84c7e4616ef7e0d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cd6603d92c4cb4b076d1b268c9e61f" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 33311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:36:07.075719 20665 mvcc.cc:205] Tried to move back new op lower bound from 7298579525890146304 to 7298579525775187968. Current Snapshot: MvccSnapshot[applied={T|T < 7298579525890146304}]
I20260619 14:36:07.076095 20664 mvcc.cc:205] Tried to move back new op lower bound from 7298579525890146304 to 7298579525775187968. Current Snapshot: MvccSnapshot[applied={T|T < 7298579525890146304}]
I20260619 14:36:07.081166 20172 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:36:07.081353 20172 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:36:07.083917 20172 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e59d3bea426f478f8f836c77176239a1: Sending DeleteTablet for 3 replicas of tablet 24aa9ae983b548539ac43760f59a5cc8
I20260619 14:36:07.084749 20566 tablet_service.cc:1514] Processing DeleteTablet for tablet 24aa9ae983b548539ac43760f59a5cc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:36:07 UTC) from {username='kudu', principal='kudu/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:37980
I20260619 14:36:07.084738 20432 tablet_service.cc:1514] Processing DeleteTablet for tablet 24aa9ae983b548539ac43760f59a5cc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:36:07 UTC) from {username='kudu', principal='kudu/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:47244
I20260619 14:36:07.084772 20298 tablet_service.cc:1514] Processing DeleteTablet for tablet 24aa9ae983b548539ac43760f59a5cc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:36:07 UTC) from {username='kudu', principal='kudu/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:54138
I20260619 14:36:07.085261 20674 tablet_replica.cc:330] T 24aa9ae983b548539ac43760f59a5cc8 P ffb58cb1737b40ad84c7e4616ef7e0d2: stopping tablet replica
I20260619 14:36:07.085265 20673 tablet_replica.cc:330] T 24aa9ae983b548539ac43760f59a5cc8 P e4cd6603d92c4cb4b076d1b268c9e61f: stopping tablet replica
I20260619 14:36:07.085601 20673 raft_consensus.cc:2229] T 24aa9ae983b548539ac43760f59a5cc8 P e4cd6603d92c4cb4b076d1b268c9e61f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:36:07.085619 20674 raft_consensus.cc:2229] T 24aa9ae983b548539ac43760f59a5cc8 P ffb58cb1737b40ad84c7e4616ef7e0d2 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:36:07.085975 20674 raft_consensus.cc:2258] T 24aa9ae983b548539ac43760f59a5cc8 P ffb58cb1737b40ad84c7e4616ef7e0d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:36:07.086005 20673 raft_consensus.cc:2258] T 24aa9ae983b548539ac43760f59a5cc8 P e4cd6603d92c4cb4b076d1b268c9e61f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:36:07.086210 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 20235
I20260619 14:36:07.088245 20672 tablet_replica.cc:330] T 24aa9ae983b548539ac43760f59a5cc8 P f6ea195ca6c14b0db819a2d864e7e728: stopping tablet replica
I20260619 14:36:07.088526 20672 raft_consensus.cc:2229] T 24aa9ae983b548539ac43760f59a5cc8 P f6ea195ca6c14b0db819a2d864e7e728 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:36:07.088825 20672 raft_consensus.cc:2258] T 24aa9ae983b548539ac43760f59a5cc8 P f6ea195ca6c14b0db819a2d864e7e728 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:36:07.089275 20673 ts_tablet_manager.cc:1905] T 24aa9ae983b548539ac43760f59a5cc8 P e4cd6603d92c4cb4b076d1b268c9e61f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:36:07.090049 20672 ts_tablet_manager.cc:1905] T 24aa9ae983b548539ac43760f59a5cc8 P f6ea195ca6c14b0db819a2d864e7e728: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:36:07.092288 20672 ts_tablet_manager.cc:1918] T 24aa9ae983b548539ac43760f59a5cc8 P f6ea195ca6c14b0db819a2d864e7e728: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:36:07.092402 20672 log.cc:1196] T 24aa9ae983b548539ac43760f59a5cc8 P f6ea195ca6c14b0db819a2d864e7e728: Deleting WAL directory at /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/ts-2/wal/wals/24aa9ae983b548539ac43760f59a5cc8
I20260619 14:36:07.092777 20672 ts_tablet_manager.cc:1939] T 24aa9ae983b548539ac43760f59a5cc8 P f6ea195ca6c14b0db819a2d864e7e728: Deleting consensus metadata
I20260619 14:36:07.092854 20673 ts_tablet_manager.cc:1918] T 24aa9ae983b548539ac43760f59a5cc8 P e4cd6603d92c4cb4b076d1b268c9e61f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:36:07.092959 20673 log.cc:1196] T 24aa9ae983b548539ac43760f59a5cc8 P e4cd6603d92c4cb4b076d1b268c9e61f: Deleting WAL directory at /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879514993456-2056-0/minicluster-data/ts-1/wal/wals/24aa9ae983b548539ac43760f59a5cc8
I20260619 14:36:07.093318 20673 ts_tablet_manager.cc:1939] T 24aa9ae983b548539ac43760f59a5cc8 P e4cd6603d92c4cb4b076d1b268c9e61f: Deleting consensus metadata
W20260619 14:36:07.093932 20158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.2.1:33007 (error 108)
I20260619 14:36:07.094144 20160 catalog_manager.cc:4841] TS f6ea195ca6c14b0db819a2d864e7e728 (127.2.2.3:39991): tablet 24aa9ae983b548539ac43760f59a5cc8 (table test-table [id=7c85d3ea190843f49bda1a93586c7043]) successfully deleted
I20260619 14:36:07.094815 20158 catalog_manager.cc:4841] TS e4cd6603d92c4cb4b076d1b268c9e61f (127.2.2.2:33311): tablet 24aa9ae983b548539ac43760f59a5cc8 (table test-table [id=7c85d3ea190843f49bda1a93586c7043]) successfully deleted
I20260619 14:36:07.094837 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 20369
I20260619 14:36:07.095016 20643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:07.094697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.1:33007 (local address 127.0.0.1:54154)
0619 14:36:07.094841 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:07.094912 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.1:33007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260619 14:36:07.095774 20158 catalog_manager.cc:4584] TS ffb58cb1737b40ad84c7e4616ef7e0d2 (127.2.2.1:33007): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 24aa9ae983b548539ac43760f59a5cc8: Network error: Client connection negotiation failed: client connection to 127.2.2.1:33007: connect: Connection refused (error 111)
I20260619 14:36:07.100476 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 20503
I20260619 14:36:07.105633 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 20141
2026-06-19T14:36:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2252 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-19T14:36:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:36:07Z Disabled control of system clock
I20260619 14:36:07.136881 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:44567
--webserver_interface=127.2.2.62
--webserver_port=0
--builtin_ntp_servers=127.2.2.20:42593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.2.62:44567
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 14:36:07.249730 20679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:36:07.250032 20679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:36:07.250069 20679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:36:07.254339 20679 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:36:07.254410 20679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:36:07.254424 20679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:36:07.254442 20679 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:36:07.254457 20679 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:36:07.259474 20679 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:42593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.2.62:44567
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:44567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.2.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:36:07.260680 20679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:36:07.262233 20679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:36:07.270778 20687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:07.270900 20679 server_base.cc:1034] running on GCE node
W20260619 14:36:07.270771 20685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:07.270816 20684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:07.271742 20679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:36:07.272851 20679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:36:07.274065 20679 hybrid_clock.cc:648] HybridClock initialized: now 1781879767274048 us; error 27 us; skew 500 ppm
I20260619 14:36:07.276058 20679 webserver.cc:460] Webserver started at http://127.2.2.62:39523/ using document root <none> and password file <none>
I20260619 14:36:07.276657 20679 fs_manager.cc:359] Metadata directory not provided
I20260619 14:36:07.276753 20679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:36:07.276994 20679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:36:07.279136 20679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/master-0/data/instance:
uuid: "b97068f8dc8441aaa1e1ffeaf8609b00"
format_stamp: "Formatted at 2026-06-19 14:36:07 on dist-test-slave-70rm"
server_key: "5517424f53254776ddd7ce3636148a8b"
server_key_iv: "e6c60a938fa0704498ba0c2865480f26"
server_key_version: "encryptionkey@0"
I20260619 14:36:07.279713 20679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/master-0/wal/instance:
uuid: "b97068f8dc8441aaa1e1ffeaf8609b00"
format_stamp: "Formatted at 2026-06-19 14:36:07 on dist-test-slave-70rm"
server_key: "5517424f53254776ddd7ce3636148a8b"
server_key_iv: "e6c60a938fa0704498ba0c2865480f26"
server_key_version: "encryptionkey@0"
I20260619 14:36:07.283354 20679 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:36:07.285782 20693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:07.287022 20679 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 14:36:07.287161 20679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/master-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/master-0/wal
uuid: "b97068f8dc8441aaa1e1ffeaf8609b00"
format_stamp: "Formatted at 2026-06-19 14:36:07 on dist-test-slave-70rm"
server_key: "5517424f53254776ddd7ce3636148a8b"
server_key_iv: "e6c60a938fa0704498ba0c2865480f26"
server_key_version: "encryptionkey@0"
I20260619 14:36:07.287283 20679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:36:07.305456 20679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:36:07.306264 20679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:36:07.306464 20679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:36:07.313469 20679 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.62:44567
I20260619 14:36:07.313503 20745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.62:44567 every 8 connection(s)
I20260619 14:36:07.314512 20679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
I20260619 14:36:07.317636 20746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:36:07.322978 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 20679
I20260619 14:36:07.323110 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/master-0/wal/instance
I20260619 14:36:07.323395 2056 external_mini_cluster.cc:1428] Setting key 7f3d6865790f6d5cf7fde41c1c3ea0a1
I20260619 14:36:07.323485 20746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b97068f8dc8441aaa1e1ffeaf8609b00: Bootstrap starting.
I20260619 14:36:07.326092 20746 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b97068f8dc8441aaa1e1ffeaf8609b00: Neither blocks nor log segments found. Creating new log.
I20260619 14:36:07.326897 20746 log.cc:824] T 00000000000000000000000000000000 P b97068f8dc8441aaa1e1ffeaf8609b00: Log is configured to *not* fsync() on all Append() calls
I20260619 14:36:07.328989 20746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b97068f8dc8441aaa1e1ffeaf8609b00: No bootstrap required, opened a new log
I20260619 14:36:07.332916 20749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:07.324801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58286 (local address 127.2.2.62:44567)
0619 14:36:07.325464 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:07.325480 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:07.325518 (+ 38us) server_negotiation.cc:407] Connection header received
0619 14:36:07.326261 (+ 743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:07.326287 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:07.326717 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:07.327062 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:07.327506 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:07.328453 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:07.329686 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:07.330219 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:07.330340 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:07.331184 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:07.331189 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:07.331201 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:36:07.331590 (+ 389us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:07.331644 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:36:07.331663 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:07.331746 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:07.331901 (+ 155us) server_negotiation.cc:299] Negotiation successful
0619 14:36:07.332187 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":129,"threads_started":1}
I20260619 14:36:07.333153 20746 raft_consensus.cc:348] T 00000000000000000000000000000000 P b97068f8dc8441aaa1e1ffeaf8609b00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b97068f8dc8441aaa1e1ffeaf8609b00" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 44567 } }
I20260619 14:36:07.333412 20746 raft_consensus.cc:374] T 00000000000000000000000000000000 P b97068f8dc8441aaa1e1ffeaf8609b00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:36:07.333490 20746 raft_consensus.cc:729] T 00000000000000000000000000000000 P b97068f8dc8441aaa1e1ffeaf8609b00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b97068f8dc8441aaa1e1ffeaf8609b00, State: Initialized, Role: FOLLOWER
I20260619 14:36:07.333981 20746 consensus_queue.cc:260] T 00000000000000000000000000000000 P b97068f8dc8441aaa1e1ffeaf8609b00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b97068f8dc8441aaa1e1ffeaf8609b00" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 44567 } }
I20260619 14:36:07.334154 20746 raft_consensus.cc:388] T 00000000000000000000000000000000 P b97068f8dc8441aaa1e1ffeaf8609b00 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:36:07.334223 20746 raft_consensus.cc:482] T 00000000000000000000000000000000 P b97068f8dc8441aaa1e1ffeaf8609b00 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:36:07.334326 20746 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b97068f8dc8441aaa1e1ffeaf8609b00 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:36:07.335371 20746 raft_consensus.cc:504] T 00000000000000000000000000000000 P b97068f8dc8441aaa1e1ffeaf8609b00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b97068f8dc8441aaa1e1ffeaf8609b00" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 44567 } }
I20260619 14:36:07.335659 20746 leader_election.cc:302] T 00000000000000000000000000000000 P b97068f8dc8441aaa1e1ffeaf8609b00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b97068f8dc8441aaa1e1ffeaf8609b00; no voters:
I20260619 14:36:07.335899 20746 leader_election.cc:288] T 00000000000000000000000000000000 P b97068f8dc8441aaa1e1ffeaf8609b00 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:36:07.336017 20751 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b97068f8dc8441aaa1e1ffeaf8609b00 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:36:07.336236 20751 raft_consensus.cc:686] T 00000000000000000000000000000000 P b97068f8dc8441aaa1e1ffeaf8609b00 [term 1 LEADER]: Becoming Leader. State: Replica: b97068f8dc8441aaa1e1ffeaf8609b00, State: Running, Role: LEADER
I20260619 14:36:07.336541 20751 consensus_queue.cc:237] T 00000000000000000000000000000000 P b97068f8dc8441aaa1e1ffeaf8609b00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b97068f8dc8441aaa1e1ffeaf8609b00" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 44567 } }
I20260619 14:36:07.337087 20746 sys_catalog.cc:564] T 00000000000000000000000000000000 P b97068f8dc8441aaa1e1ffeaf8609b00 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:36:07.338147 20752 sys_catalog.cc:455] T 00000000000000000000000000000000 P b97068f8dc8441aaa1e1ffeaf8609b00 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b97068f8dc8441aaa1e1ffeaf8609b00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b97068f8dc8441aaa1e1ffeaf8609b00" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 44567 } } }
I20260619 14:36:07.338286 20752 sys_catalog.cc:458] T 00000000000000000000000000000000 P b97068f8dc8441aaa1e1ffeaf8609b00 [sys.catalog]: This master's current role is: LEADER
I20260619 14:36:07.338565 20753 sys_catalog.cc:455] T 00000000000000000000000000000000 P b97068f8dc8441aaa1e1ffeaf8609b00 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b97068f8dc8441aaa1e1ffeaf8609b00. Latest consensus state: current_term: 1 leader_uuid: "b97068f8dc8441aaa1e1ffeaf8609b00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b97068f8dc8441aaa1e1ffeaf8609b00" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 44567 } } }
I20260619 14:36:07.338668 20753 sys_catalog.cc:458] T 00000000000000000000000000000000 P b97068f8dc8441aaa1e1ffeaf8609b00 [sys.catalog]: This master's current role is: LEADER
I20260619 14:36:07.338680 20760 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:36:07.341827 20760 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:36:07.347783 20760 catalog_manager.cc:1347] Generated new cluster ID: ffe84e35e7e24543932c936c5add0c6d
I20260619 14:36:07.347855 20760 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:36:07.393726 20760 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:36:07.394767 20760 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:36:07.402661 20760 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b97068f8dc8441aaa1e1ffeaf8609b00: Generated new TSK 0
I20260619 14:36:07.403385 20760 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:36:07.409408 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.1:0
--local_ip_for_outbound_sockets=127.2.2.1
--webserver_interface=127.2.2.1
--webserver_port=0
--tserver_master_addrs=127.2.2.62:44567
--builtin_ntp_servers=127.2.2.20:42593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 14:36:07.519968 20770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:36:07.520294 20770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:36:07.520376 20770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:36:07.524435 20770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:36:07.524528 20770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:36:07.524633 20770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.1
I20260619 14:36:07.529429 20770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:42593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.2.1
--webserver_port=0
--tserver_master_addrs=127.2.2.62:44567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.1
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:36:07.530597 20770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:36:07.532047 20770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:36:07.542179 20778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:07.542179 20776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:07.542174 20775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:07.542529 20770 server_base.cc:1034] running on GCE node
I20260619 14:36:07.543012 20770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:36:07.543715 20770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:36:07.544885 20770 hybrid_clock.cc:648] HybridClock initialized: now 1781879767544866 us; error 33 us; skew 500 ppm
I20260619 14:36:07.546850 20770 webserver.cc:460] Webserver started at http://127.2.2.1:41791/ using document root <none> and password file <none>
I20260619 14:36:07.547387 20770 fs_manager.cc:359] Metadata directory not provided
I20260619 14:36:07.547444 20770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:36:07.547609 20770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:36:07.549629 20770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/ts-0/data/instance:
uuid: "e6522d4891b349c68e1bb1a298a8ddeb"
format_stamp: "Formatted at 2026-06-19 14:36:07 on dist-test-slave-70rm"
server_key: "6abc97463f2ea21c2e334b9eee03ca24"
server_key_iv: "d9eadf8e2f78df9d78b773eeb00feb38"
server_key_version: "encryptionkey@0"
I20260619 14:36:07.550190 20770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/ts-0/wal/instance:
uuid: "e6522d4891b349c68e1bb1a298a8ddeb"
format_stamp: "Formatted at 2026-06-19 14:36:07 on dist-test-slave-70rm"
server_key: "6abc97463f2ea21c2e334b9eee03ca24"
server_key_iv: "d9eadf8e2f78df9d78b773eeb00feb38"
server_key_version: "encryptionkey@0"
I20260619 14:36:07.553745 20770 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260619 14:36:07.556274 20784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:07.557483 20770 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260619 14:36:07.557612 20770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/ts-0/wal
uuid: "e6522d4891b349c68e1bb1a298a8ddeb"
format_stamp: "Formatted at 2026-06-19 14:36:07 on dist-test-slave-70rm"
server_key: "6abc97463f2ea21c2e334b9eee03ca24"
server_key_iv: "d9eadf8e2f78df9d78b773eeb00feb38"
server_key_version: "encryptionkey@0"
I20260619 14:36:07.557726 20770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:36:07.575723 20770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:36:07.576416 20770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:36:07.576608 20770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:36:07.577191 20770 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:36:07.578150 20770 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:36:07.578219 20770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:07.578286 20770 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:36:07.578327 20770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:07.588994 20770 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.1:33773
I20260619 14:36:07.589011 20897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.1:33773 every 8 connection(s)
I20260619 14:36:07.589943 20770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
I20260619 14:36:07.595929 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 20770
I20260619 14:36:07.596025 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/ts-0/wal/instance
I20260619 14:36:07.596318 2056 external_mini_cluster.cc:1428] Setting key 4096bd6c15048836041961b4c429e00e
I20260619 14:36:07.598526 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.2:0
--local_ip_for_outbound_sockets=127.2.2.2
--webserver_interface=127.2.2.2
--webserver_port=0
--tserver_master_addrs=127.2.2.62:44567
--builtin_ntp_servers=127.2.2.20:42593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 14:36:07.600270 20749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:07.591904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:38175 (local address 127.2.2.62:44567)
0619 14:36:07.592043 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:07.592047 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:07.593207 (+ 1160us) server_negotiation.cc:407] Connection header received
0619 14:36:07.594780 (+ 1573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:07.594784 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:07.594847 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:07.594961 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:07.595895 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:07.596617 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:07.598096 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:07.598398 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:07.598494 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:07.599800 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:07.599803 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:07.599805 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:36:07.599816 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:07.599836 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:36:07.599844 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:07.599898 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:07.600094 (+ 196us) server_negotiation.cc:299] Negotiation successful
0619 14:36:07.600146 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 14:36:07.601583 20900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:07.592346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:44567 (local address 127.2.2.1:38175)
0619 14:36:07.593033 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:07.593079 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 14:36:07.594543 (+ 1464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:36:07.595140 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:36:07.595147 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:36:07.595234 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:36:07.595742 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:07.595756 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:07.596778 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:07.596781 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:07.597960 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:07.597971 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:07.598510 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:07.598570 (+ 60us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:07.598812 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:07.599479 (+ 667us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:36:07.599484 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:36:07.599543 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:36:07.599633 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:36:07.599927 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:36:07.599939 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:36:07.599948 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 14:36:07.600095 (+ 147us) client_negotiation.cc:240] Negotiation successful
0619 14:36:07.600525 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":113,"threads_started":1}
I20260619 14:36:07.602897 20898 heartbeater.cc:347] Connected to a master server at 127.2.2.62:44567
I20260619 14:36:07.603176 20898 heartbeater.cc:464] Registering TS with master...
I20260619 14:36:07.603885 20898 heartbeater.cc:511] Master 127.2.2.62:44567 requested a full tablet report, sending...
I20260619 14:36:07.605705 20710 ts_manager.cc:194] Registered new tserver with Master: e6522d4891b349c68e1bb1a298a8ddeb (127.2.2.1:33773)
I20260619 14:36:07.607507 20710 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.2.1:38175
W20260619 14:36:07.712464 20901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:36:07.712738 20901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:36:07.712777 20901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:36:07.716856 20901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:36:07.716917 20901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:36:07.716971 20901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.2
I20260619 14:36:07.721650 20901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:42593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.2.2
--webserver_port=0
--tserver_master_addrs=127.2.2.62:44567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.2
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:36:07.722782 20901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:36:07.724125 20901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:36:07.733817 20907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:07.733798 20906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:07.734076 20901 server_base.cc:1034] running on GCE node
W20260619 14:36:07.733820 20909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:07.734575 20901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:36:07.735231 20901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:36:07.736426 20901 hybrid_clock.cc:648] HybridClock initialized: now 1781879767736407 us; error 31 us; skew 500 ppm
I20260619 14:36:07.738495 20901 webserver.cc:460] Webserver started at http://127.2.2.2:44483/ using document root <none> and password file <none>
I20260619 14:36:07.739157 20901 fs_manager.cc:359] Metadata directory not provided
I20260619 14:36:07.739250 20901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:36:07.739478 20901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:36:07.741611 20901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/ts-1/data/instance:
uuid: "574d7cda29d84680ac253bc2de2a2cfe"
format_stamp: "Formatted at 2026-06-19 14:36:07 on dist-test-slave-70rm"
server_key: "1699bf6da476123ab4bb29cadcbe72f6"
server_key_iv: "2dda488e784dd912425b038ce2367869"
server_key_version: "encryptionkey@0"
I20260619 14:36:07.742250 20901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/ts-1/wal/instance:
uuid: "574d7cda29d84680ac253bc2de2a2cfe"
format_stamp: "Formatted at 2026-06-19 14:36:07 on dist-test-slave-70rm"
server_key: "1699bf6da476123ab4bb29cadcbe72f6"
server_key_iv: "2dda488e784dd912425b038ce2367869"
server_key_version: "encryptionkey@0"
I20260619 14:36:07.745963 20901 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 14:36:07.748519 20915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:07.749770 20901 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:36:07.749917 20901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/ts-1/wal
uuid: "574d7cda29d84680ac253bc2de2a2cfe"
format_stamp: "Formatted at 2026-06-19 14:36:07 on dist-test-slave-70rm"
server_key: "1699bf6da476123ab4bb29cadcbe72f6"
server_key_iv: "2dda488e784dd912425b038ce2367869"
server_key_version: "encryptionkey@0"
I20260619 14:36:07.750037 20901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:36:07.771946 20901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:36:07.772686 20901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:36:07.772897 20901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:36:07.773506 20901 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:36:07.774508 20901 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:36:07.774581 20901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:07.774654 20901 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:36:07.774695 20901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:07.783677 20901 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.2:42003
I20260619 14:36:07.783700 21028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.2:42003 every 8 connection(s)
I20260619 14:36:07.784644 20901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
I20260619 14:36:07.785622 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 20901
I20260619 14:36:07.785746 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/ts-1/wal/instance
I20260619 14:36:07.786037 2056 external_mini_cluster.cc:1428] Setting key 3cb395478e5c38109e9103e0f69458dc
I20260619 14:36:07.788326 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.3:0
--local_ip_for_outbound_sockets=127.2.2.3
--webserver_interface=127.2.2.3
--webserver_port=0
--tserver_master_addrs=127.2.2.62:44567
--builtin_ntp_servers=127.2.2.20:42593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 14:36:07.796060 20749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:07.786861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:46577 (local address 127.2.2.62:44567)
0619 14:36:07.786996 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:07.787000 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:07.788307 (+ 1307us) server_negotiation.cc:407] Connection header received
0619 14:36:07.790405 (+ 2098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:07.790411 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:07.790478 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:07.790596 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:07.791559 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:07.792466 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:07.793672 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:07.794105 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:07.794269 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:07.795393 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:07.795399 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:07.795405 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:36:07.795423 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:07.795450 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:36:07.795464 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:07.795556 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:07.795766 (+ 210us) server_negotiation.cc:299] Negotiation successful
0619 14:36:07.795845 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 14:36:07.796890 21031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:07.787323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:44567 (local address 127.2.2.2:46577)
0619 14:36:07.788153 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:07.788215 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 14:36:07.790191 (+ 1976us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:36:07.790810 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:36:07.790817 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:36:07.790911 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:36:07.791416 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:07.791429 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:07.792616 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:07.792619 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:07.793532 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:07.793542 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:07.794234 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:07.794285 (+ 51us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:07.794435 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:07.795124 (+ 689us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:36:07.795126 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:36:07.795169 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:36:07.795234 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:36:07.795590 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:36:07.795598 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:36:07.795603 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 14:36:07.795740 (+ 137us) client_negotiation.cc:240] Negotiation successful
0619 14:36:07.795978 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":136,"threads_started":1}
I20260619 14:36:07.798164 21029 heartbeater.cc:347] Connected to a master server at 127.2.2.62:44567
I20260619 14:36:07.798411 21029 heartbeater.cc:464] Registering TS with master...
I20260619 14:36:07.798986 21029 heartbeater.cc:511] Master 127.2.2.62:44567 requested a full tablet report, sending...
I20260619 14:36:07.800060 20710 ts_manager.cc:194] Registered new tserver with Master: 574d7cda29d84680ac253bc2de2a2cfe (127.2.2.2:42003)
I20260619 14:36:07.800652 20710 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.2.2:46577
W20260619 14:36:07.903116 21032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:36:07.903383 21032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:36:07.903461 21032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:36:07.907505 21032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:36:07.907569 21032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:36:07.907673 21032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.3
I20260619 14:36:07.912462 21032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:42593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.2.3
--webserver_port=0
--tserver_master_addrs=127.2.2.62:44567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.3
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:36:07.913518 21032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:36:07.914938 21032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:36:07.924752 21037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:07.924752 21040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:07.924769 21038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:07.924793 21032 server_base.cc:1034] running on GCE node
I20260619 14:36:07.925432 21032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:36:07.926093 21032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:36:07.927318 21032 hybrid_clock.cc:648] HybridClock initialized: now 1781879767927295 us; error 39 us; skew 500 ppm
I20260619 14:36:07.929232 21032 webserver.cc:460] Webserver started at http://127.2.2.3:42847/ using document root <none> and password file <none>
I20260619 14:36:07.929843 21032 fs_manager.cc:359] Metadata directory not provided
I20260619 14:36:07.929908 21032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:36:07.930158 21032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:36:07.932214 21032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/ts-2/data/instance:
uuid: "117eb9b3303241de8c51cfbedb5c228e"
format_stamp: "Formatted at 2026-06-19 14:36:07 on dist-test-slave-70rm"
server_key: "db9ba241b072c444c4ad8e60a52fab58"
server_key_iv: "8797f69ccaafcc39794eea53792fe43b"
server_key_version: "encryptionkey@0"
I20260619 14:36:07.932801 21032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/ts-2/wal/instance:
uuid: "117eb9b3303241de8c51cfbedb5c228e"
format_stamp: "Formatted at 2026-06-19 14:36:07 on dist-test-slave-70rm"
server_key: "db9ba241b072c444c4ad8e60a52fab58"
server_key_iv: "8797f69ccaafcc39794eea53792fe43b"
server_key_version: "encryptionkey@0"
I20260619 14:36:07.936457 21032 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260619 14:36:07.939044 21046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:07.940363 21032 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:36:07.940498 21032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/ts-2/wal
uuid: "117eb9b3303241de8c51cfbedb5c228e"
format_stamp: "Formatted at 2026-06-19 14:36:07 on dist-test-slave-70rm"
server_key: "db9ba241b072c444c4ad8e60a52fab58"
server_key_iv: "8797f69ccaafcc39794eea53792fe43b"
server_key_version: "encryptionkey@0"
I20260619 14:36:07.940614 21032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:36:07.957302 21032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:36:07.958030 21032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:36:07.958257 21032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:36:07.958856 21032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:36:07.959797 21032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:36:07.959844 21032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:07.959913 21032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:36:07.959951 21032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:07.968665 21032 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.3:33039
I20260619 14:36:07.968735 21159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.3:33039 every 8 connection(s)
I20260619 14:36:07.969638 21032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
I20260619 14:36:07.975389 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 21032
I20260619 14:36:07.975477 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/ts-2/wal/instance
I20260619 14:36:07.975751 2056 external_mini_cluster.cc:1428] Setting key f1b1886b9a58ee6eee87a44a8f058172
I20260619 14:36:07.979148 20749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:07.971492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.3:57825 (local address 127.2.2.62:44567)
0619 14:36:07.971638 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:07.971642 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:07.972696 (+ 1054us) server_negotiation.cc:407] Connection header received
0619 14:36:07.974024 (+ 1328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:07.974026 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:07.974075 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:07.974219 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:07.975102 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:07.975701 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:07.977129 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:07.977371 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:07.977485 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:07.978631 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:07.978635 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:07.978638 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:36:07.978652 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:07.978679 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:36:07.978688 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:07.978776 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:07.978938 (+ 162us) server_negotiation.cc:299] Negotiation successful
0619 14:36:07.979005 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 14:36:07.979987 21162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:07.971875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:44567 (local address 127.2.2.3:57825)
0619 14:36:07.972535 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:07.972576 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 14:36:07.973830 (+ 1254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:36:07.974392 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:36:07.974400 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:36:07.974489 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:36:07.974976 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:07.974990 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:07.975841 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:07.975846 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:07.977007 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:07.977016 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:07.977471 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:07.977525 (+ 54us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:07.977701 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:07.978373 (+ 672us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:36:07.978375 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:36:07.978416 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:36:07.978480 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:36:07.978776 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:36:07.978787 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:36:07.978795 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 14:36:07.978927 (+ 132us) client_negotiation.cc:240] Negotiation successful
0619 14:36:07.979214 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":119,"threads_started":1}
I20260619 14:36:07.981037 21160 heartbeater.cc:347] Connected to a master server at 127.2.2.62:44567
I20260619 14:36:07.981264 21160 heartbeater.cc:464] Registering TS with master...
I20260619 14:36:07.981734 21160 heartbeater.cc:511] Master 127.2.2.62:44567 requested a full tablet report, sending...
I20260619 14:36:07.982700 20710 ts_manager.cc:194] Registered new tserver with Master: 117eb9b3303241de8c51cfbedb5c228e (127.2.2.3:33039)
I20260619 14:36:07.983289 20710 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.2.3:57825
I20260619 14:36:07.989632 2056 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:36:07.995025 20749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:07.991518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58298 (local address 127.2.2.62:44567)
0619 14:36:07.991630 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:07.991634 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:07.991802 (+ 168us) server_negotiation.cc:407] Connection header received
0619 14:36:07.991922 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:07.991924 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:07.991963 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:07.992029 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:07.992284 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:07.992780 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:07.993544 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:07.993759 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:07.993831 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:07.994641 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:07.994644 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:07.994646 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:36:07.994657 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:07.994674 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:36:07.994683 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:07.994789 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:07.994886 (+ 97us) server_negotiation.cc:299] Negotiation successful
0619 14:36:07.994918 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 14:36:07.997931 20710 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58298:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:36:08.000960 20710 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:36:08.015112 21173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:08.009259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60860 (local address 127.2.2.1:33773)
0619 14:36:08.009712 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:08.009724 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:08.009749 (+ 25us) server_negotiation.cc:407] Connection header received
0619 14:36:08.009958 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:08.009966 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:08.010227 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:08.010376 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:36:08.010896 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:08.012069 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:08.013573 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:08.014355 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:08.014468 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:08.014675 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:08.014828 (+ 153us) server_negotiation.cc:299] Negotiation successful
0619 14:36:08.014936 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":124,"threads_started":1}
I20260619 14:36:08.015237 21172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:08.009198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.2:42003 (local address 127.0.0.1:35146)
0619 14:36:08.010031 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:08.010044 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:36:08.010142 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:36:08.011191 (+ 1049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:36:08.011193 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:36:08.011210 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:36:08.011311 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:08.011321 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:08.012450 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:08.012454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:08.013713 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:08.013725 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:08.014819 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:08.014823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:08.014916 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:08.014938 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 14:36:08.014988 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 14:36:08.015059 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774,"mutex_wait_us":32,"thread_start_us":86,"threads_started":1}
I20260619 14:36:08.015090 21171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:08.009112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.1:33773 (local address 127.0.0.1:60860)
0619 14:36:08.009561 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:08.009614 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 14:36:08.009754 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:36:08.010563 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:36:08.010570 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:36:08.010596 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:36:08.010748 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:08.010758 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:08.012196 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:08.012200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:08.013449 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:08.013458 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:08.014487 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:08.014490 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:08.014618 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:08.014661 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 14:36:08.014786 (+ 125us) client_negotiation.cc:240] Negotiation successful
0619 14:36:08.014909 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":102,"threads_started":1}
I20260619 14:36:08.015681 21174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:08.009885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.3:33039 (local address 127.0.0.1:59412)
0619 14:36:08.010707 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:08.010721 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:36:08.010826 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:36:08.011357 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:36:08.011363 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:36:08.011384 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:36:08.011460 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:08.011466 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:08.012850 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:08.012853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:08.013978 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:08.013985 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:08.015239 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:08.015245 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:08.015344 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:08.015369 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 14:36:08.015451 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 14:36:08.015531 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":759,"mutex_wait_us":30,"thread_start_us":91,"threads_started":1}
I20260619 14:36:08.015954 21176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:08.010349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35146 (local address 127.2.2.2:42003)
0619 14:36:08.010786 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:08.010797 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:08.010820 (+ 23us) server_negotiation.cc:407] Connection header received
0619 14:36:08.010896 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:08.010903 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:08.011065 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:08.011215 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:36:08.011440 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:08.012259 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:08.013857 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:08.014644 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:08.015176 (+ 532us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:08.015365 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:08.015676 (+ 311us) server_negotiation.cc:299] Negotiation successful
0619 14:36:08.015801 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":102,"threads_started":1}
I20260619 14:36:08.016139 21175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:08.010336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59412 (local address 127.2.2.3:33039)
0619 14:36:08.010717 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:08.010729 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:08.010826 (+ 97us) server_negotiation.cc:407] Connection header received
0619 14:36:08.010938 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:08.010945 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:08.011137 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:08.011296 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:36:08.011580 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:08.012707 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:08.014116 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:08.014891 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:08.014989 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:08.015179 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:08.015876 (+ 697us) server_negotiation.cc:299] Negotiation successful
0619 14:36:08.015976 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":90,"threads_started":1}
I20260619 14:36:08.018529 20963 tablet_service.cc:1467] Processing CreateTablet for tablet 8ca0c1466fab4ff0927822abe5ce810a (DEFAULT_TABLE table=test-table [id=254fc5ecf6cc498c90929c15652dcba5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:36:08.018541 20829 tablet_service.cc:1467] Processing CreateTablet for tablet 8ca0c1466fab4ff0927822abe5ce810a (DEFAULT_TABLE table=test-table [id=254fc5ecf6cc498c90929c15652dcba5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:36:08.018541 21094 tablet_service.cc:1467] Processing CreateTablet for tablet 8ca0c1466fab4ff0927822abe5ce810a (DEFAULT_TABLE table=test-table [id=254fc5ecf6cc498c90929c15652dcba5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:36:08.019526 20963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ca0c1466fab4ff0927822abe5ce810a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:36:08.019551 21094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ca0c1466fab4ff0927822abe5ce810a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:36:08.019552 20829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ca0c1466fab4ff0927822abe5ce810a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:36:08.026000 21177 tablet_bootstrap.cc:492] T 8ca0c1466fab4ff0927822abe5ce810a P 574d7cda29d84680ac253bc2de2a2cfe: Bootstrap starting.
I20260619 14:36:08.027608 21178 tablet_bootstrap.cc:492] T 8ca0c1466fab4ff0927822abe5ce810a P 117eb9b3303241de8c51cfbedb5c228e: Bootstrap starting.
I20260619 14:36:08.027822 21179 tablet_bootstrap.cc:492] T 8ca0c1466fab4ff0927822abe5ce810a P e6522d4891b349c68e1bb1a298a8ddeb: Bootstrap starting.
I20260619 14:36:08.028631 21177 tablet_bootstrap.cc:654] T 8ca0c1466fab4ff0927822abe5ce810a P 574d7cda29d84680ac253bc2de2a2cfe: Neither blocks nor log segments found. Creating new log.
I20260619 14:36:08.029542 21177 log.cc:824] T 8ca0c1466fab4ff0927822abe5ce810a P 574d7cda29d84680ac253bc2de2a2cfe: Log is configured to *not* fsync() on all Append() calls
I20260619 14:36:08.029726 21179 tablet_bootstrap.cc:654] T 8ca0c1466fab4ff0927822abe5ce810a P e6522d4891b349c68e1bb1a298a8ddeb: Neither blocks nor log segments found. Creating new log.
I20260619 14:36:08.029826 21178 tablet_bootstrap.cc:654] T 8ca0c1466fab4ff0927822abe5ce810a P 117eb9b3303241de8c51cfbedb5c228e: Neither blocks nor log segments found. Creating new log.
I20260619 14:36:08.030628 21179 log.cc:824] T 8ca0c1466fab4ff0927822abe5ce810a P e6522d4891b349c68e1bb1a298a8ddeb: Log is configured to *not* fsync() on all Append() calls
I20260619 14:36:08.030642 21178 log.cc:824] T 8ca0c1466fab4ff0927822abe5ce810a P 117eb9b3303241de8c51cfbedb5c228e: Log is configured to *not* fsync() on all Append() calls
I20260619 14:36:08.032400 21177 tablet_bootstrap.cc:492] T 8ca0c1466fab4ff0927822abe5ce810a P 574d7cda29d84680ac253bc2de2a2cfe: No bootstrap required, opened a new log
I20260619 14:36:08.032550 21178 tablet_bootstrap.cc:492] T 8ca0c1466fab4ff0927822abe5ce810a P 117eb9b3303241de8c51cfbedb5c228e: No bootstrap required, opened a new log
I20260619 14:36:08.032618 21177 ts_tablet_manager.cc:1397] T 8ca0c1466fab4ff0927822abe5ce810a P 574d7cda29d84680ac253bc2de2a2cfe: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260619 14:36:08.032707 21178 ts_tablet_manager.cc:1397] T 8ca0c1466fab4ff0927822abe5ce810a P 117eb9b3303241de8c51cfbedb5c228e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:36:08.032945 21179 tablet_bootstrap.cc:492] T 8ca0c1466fab4ff0927822abe5ce810a P e6522d4891b349c68e1bb1a298a8ddeb: No bootstrap required, opened a new log
I20260619 14:36:08.033154 21179 ts_tablet_manager.cc:1397] T 8ca0c1466fab4ff0927822abe5ce810a P e6522d4891b349c68e1bb1a298a8ddeb: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260619 14:36:08.036319 21179 raft_consensus.cc:348] T 8ca0c1466fab4ff0927822abe5ce810a P e6522d4891b349c68e1bb1a298a8ddeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6522d4891b349c68e1bb1a298a8ddeb" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 33773 } } peers { permanent_uuid: "574d7cda29d84680ac253bc2de2a2cfe" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 42003 } } peers { permanent_uuid: "117eb9b3303241de8c51cfbedb5c228e" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 33039 } }
I20260619 14:36:08.036608 21179 raft_consensus.cc:374] T 8ca0c1466fab4ff0927822abe5ce810a P e6522d4891b349c68e1bb1a298a8ddeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:36:08.036669 21179 raft_consensus.cc:729] T 8ca0c1466fab4ff0927822abe5ce810a P e6522d4891b349c68e1bb1a298a8ddeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6522d4891b349c68e1bb1a298a8ddeb, State: Initialized, Role: FOLLOWER
I20260619 14:36:08.036895 21177 raft_consensus.cc:348] T 8ca0c1466fab4ff0927822abe5ce810a P 574d7cda29d84680ac253bc2de2a2cfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6522d4891b349c68e1bb1a298a8ddeb" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 33773 } } peers { permanent_uuid: "574d7cda29d84680ac253bc2de2a2cfe" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 42003 } } peers { permanent_uuid: "117eb9b3303241de8c51cfbedb5c228e" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 33039 } }
I20260619 14:36:08.036895 21178 raft_consensus.cc:348] T 8ca0c1466fab4ff0927822abe5ce810a P 117eb9b3303241de8c51cfbedb5c228e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6522d4891b349c68e1bb1a298a8ddeb" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 33773 } } peers { permanent_uuid: "574d7cda29d84680ac253bc2de2a2cfe" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 42003 } } peers { permanent_uuid: "117eb9b3303241de8c51cfbedb5c228e" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 33039 } }
I20260619 14:36:08.037163 21178 raft_consensus.cc:374] T 8ca0c1466fab4ff0927822abe5ce810a P 117eb9b3303241de8c51cfbedb5c228e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:36:08.037163 21177 raft_consensus.cc:374] T 8ca0c1466fab4ff0927822abe5ce810a P 574d7cda29d84680ac253bc2de2a2cfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:36:08.037218 21178 raft_consensus.cc:729] T 8ca0c1466fab4ff0927822abe5ce810a P 117eb9b3303241de8c51cfbedb5c228e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 117eb9b3303241de8c51cfbedb5c228e, State: Initialized, Role: FOLLOWER
I20260619 14:36:08.037218 21177 raft_consensus.cc:729] T 8ca0c1466fab4ff0927822abe5ce810a P 574d7cda29d84680ac253bc2de2a2cfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 574d7cda29d84680ac253bc2de2a2cfe, State: Initialized, Role: FOLLOWER
I20260619 14:36:08.037184 21179 consensus_queue.cc:260] T 8ca0c1466fab4ff0927822abe5ce810a P e6522d4891b349c68e1bb1a298a8ddeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6522d4891b349c68e1bb1a298a8ddeb" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 33773 } } peers { permanent_uuid: "574d7cda29d84680ac253bc2de2a2cfe" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 42003 } } peers { permanent_uuid: "117eb9b3303241de8c51cfbedb5c228e" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 33039 } }
I20260619 14:36:08.037655 21178 consensus_queue.cc:260] T 8ca0c1466fab4ff0927822abe5ce810a P 117eb9b3303241de8c51cfbedb5c228e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6522d4891b349c68e1bb1a298a8ddeb" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 33773 } } peers { permanent_uuid: "574d7cda29d84680ac253bc2de2a2cfe" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 42003 } } peers { permanent_uuid: "117eb9b3303241de8c51cfbedb5c228e" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 33039 } }
I20260619 14:36:08.037655 21177 consensus_queue.cc:260] T 8ca0c1466fab4ff0927822abe5ce810a P 574d7cda29d84680ac253bc2de2a2cfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6522d4891b349c68e1bb1a298a8ddeb" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 33773 } } peers { permanent_uuid: "574d7cda29d84680ac253bc2de2a2cfe" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 42003 } } peers { permanent_uuid: "117eb9b3303241de8c51cfbedb5c228e" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 33039 } }
I20260619 14:36:08.038362 21179 ts_tablet_manager.cc:1428] T 8ca0c1466fab4ff0927822abe5ce810a P e6522d4891b349c68e1bb1a298a8ddeb: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260619 14:36:08.038672 21178 ts_tablet_manager.cc:1428] T 8ca0c1466fab4ff0927822abe5ce810a P 117eb9b3303241de8c51cfbedb5c228e: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 14:36:08.038672 21177 ts_tablet_manager.cc:1428] T 8ca0c1466fab4ff0927822abe5ce810a P 574d7cda29d84680ac253bc2de2a2cfe: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260619 14:36:08.038769 20898 heartbeater.cc:503] Master 127.2.2.62:44567 was elected leader, sending a full tablet report...
I20260619 14:36:08.038988 21029 heartbeater.cc:503] Master 127.2.2.62:44567 was elected leader, sending a full tablet report...
I20260619 14:36:08.038988 21160 heartbeater.cc:503] Master 127.2.2.62:44567 was elected leader, sending a full tablet report...
W20260619 14:36:08.091138 20899 tablet.cc:2369] T 8ca0c1466fab4ff0927822abe5ce810a P e6522d4891b349c68e1bb1a298a8ddeb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:36:08.217903 21183 raft_consensus.cc:482] T 8ca0c1466fab4ff0927822abe5ce810a P e6522d4891b349c68e1bb1a298a8ddeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:36:08.218192 21183 raft_consensus.cc:504] T 8ca0c1466fab4ff0927822abe5ce810a P e6522d4891b349c68e1bb1a298a8ddeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6522d4891b349c68e1bb1a298a8ddeb" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 33773 } } peers { permanent_uuid: "574d7cda29d84680ac253bc2de2a2cfe" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 42003 } } peers { permanent_uuid: "117eb9b3303241de8c51cfbedb5c228e" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 33039 } }
I20260619 14:36:08.219337 21183 leader_election.cc:288] T 8ca0c1466fab4ff0927822abe5ce810a P e6522d4891b349c68e1bb1a298a8ddeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 574d7cda29d84680ac253bc2de2a2cfe (127.2.2.2:42003), 117eb9b3303241de8c51cfbedb5c228e (127.2.2.3:33039)
W20260619 14:36:08.220641 21161 tablet.cc:2369] T 8ca0c1466fab4ff0927822abe5ce810a P 117eb9b3303241de8c51cfbedb5c228e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:36:08.223909 21176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:08.219769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:46841 (local address 127.2.2.2:42003)
0619 14:36:08.219936 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:08.219941 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:08.220069 (+ 128us) server_negotiation.cc:407] Connection header received
0619 14:36:08.220239 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:08.220242 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:08.220298 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:08.220379 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:36:08.220774 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:08.221298 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:08.222808 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:08.223392 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:08.223496 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:08.223561 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:08.223713 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 14:36:08.223811 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 14:36:08.223929 21186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:08.219663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.2:42003 (local address 127.2.2.1:46841)
0619 14:36:08.219969 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:08.219987 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:36:08.220128 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:36:08.220419 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:36:08.220423 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:36:08.220451 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:36:08.220611 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:08.220618 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:08.221443 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:08.221447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:08.222691 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:08.222700 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:08.223505 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:08.223507 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:08.223590 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:08.223609 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 14:36:08.223666 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 14:36:08.223755 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":1,"spinlock_wait_cycles":37632,"thread_start_us":119,"threads_started":1}
I20260619 14:36:08.224611 20983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca0c1466fab4ff0927822abe5ce810a" candidate_uuid: "e6522d4891b349c68e1bb1a298a8ddeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "574d7cda29d84680ac253bc2de2a2cfe" is_pre_election: true
I20260619 14:36:08.224963 20983 raft_consensus.cc:2454] T 8ca0c1466fab4ff0927822abe5ce810a P 574d7cda29d84680ac253bc2de2a2cfe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6522d4891b349c68e1bb1a298a8ddeb in term 0.
I20260619 14:36:08.225070 21187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:08.220081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.3:33039 (local address 127.2.2.1:34959)
0619 14:36:08.220407 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:08.220420 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:36:08.220530 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:36:08.221392 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:36:08.221396 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:36:08.221419 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:36:08.221526 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:08.221531 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:08.222506 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:08.222508 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:08.223640 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:08.223667 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:08.224591 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:08.224594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:08.224675 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:08.224698 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 14:36:08.224823 (+ 125us) client_negotiation.cc:240] Negotiation successful
0619 14:36:08.224903 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":31,"spinlock_wait_cycles":768,"thread_start_us":68,"threads_started":1}
I20260619 14:36:08.225077 21175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:08.220190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:34959 (local address 127.2.2.3:33039)
0619 14:36:08.220374 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:08.220381 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:08.221189 (+ 808us) server_negotiation.cc:407] Connection header received
0619 14:36:08.221241 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:08.221243 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:08.221281 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:08.221346 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:36:08.221630 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:08.222403 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:08.223794 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:08.224404 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:08.224515 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:08.224570 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:08.224844 (+ 274us) server_negotiation.cc:299] Negotiation successful
0619 14:36:08.224923 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 14:36:08.225518 20786 leader_election.cc:302] T 8ca0c1466fab4ff0927822abe5ce810a P e6522d4891b349c68e1bb1a298a8ddeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 574d7cda29d84680ac253bc2de2a2cfe, e6522d4891b349c68e1bb1a298a8ddeb; no voters:
I20260619 14:36:08.225737 21114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca0c1466fab4ff0927822abe5ce810a" candidate_uuid: "e6522d4891b349c68e1bb1a298a8ddeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "117eb9b3303241de8c51cfbedb5c228e" is_pre_election: true
I20260619 14:36:08.225836 21183 raft_consensus.cc:2781] T 8ca0c1466fab4ff0927822abe5ce810a P e6522d4891b349c68e1bb1a298a8ddeb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:36:08.225935 21183 raft_consensus.cc:482] T 8ca0c1466fab4ff0927822abe5ce810a P e6522d4891b349c68e1bb1a298a8ddeb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:36:08.226003 21114 raft_consensus.cc:2454] T 8ca0c1466fab4ff0927822abe5ce810a P 117eb9b3303241de8c51cfbedb5c228e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6522d4891b349c68e1bb1a298a8ddeb in term 0.
I20260619 14:36:08.226023 21183 raft_consensus.cc:3037] T 8ca0c1466fab4ff0927822abe5ce810a P e6522d4891b349c68e1bb1a298a8ddeb [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:36:08.227416 21183 raft_consensus.cc:504] T 8ca0c1466fab4ff0927822abe5ce810a P e6522d4891b349c68e1bb1a298a8ddeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6522d4891b349c68e1bb1a298a8ddeb" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 33773 } } peers { permanent_uuid: "574d7cda29d84680ac253bc2de2a2cfe" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 42003 } } peers { permanent_uuid: "117eb9b3303241de8c51cfbedb5c228e" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 33039 } }
I20260619 14:36:08.227787 21183 leader_election.cc:288] T 8ca0c1466fab4ff0927822abe5ce810a P e6522d4891b349c68e1bb1a298a8ddeb [CANDIDATE]: Term 1 election: Requested vote from peers 574d7cda29d84680ac253bc2de2a2cfe (127.2.2.2:42003), 117eb9b3303241de8c51cfbedb5c228e (127.2.2.3:33039)
I20260619 14:36:08.228209 20983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca0c1466fab4ff0927822abe5ce810a" candidate_uuid: "e6522d4891b349c68e1bb1a298a8ddeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "574d7cda29d84680ac253bc2de2a2cfe"
I20260619 14:36:08.228266 21114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca0c1466fab4ff0927822abe5ce810a" candidate_uuid: "e6522d4891b349c68e1bb1a298a8ddeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "117eb9b3303241de8c51cfbedb5c228e"
I20260619 14:36:08.228375 21114 raft_consensus.cc:3037] T 8ca0c1466fab4ff0927822abe5ce810a P 117eb9b3303241de8c51cfbedb5c228e [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:36:08.228359 20983 raft_consensus.cc:3037] T 8ca0c1466fab4ff0927822abe5ce810a P 574d7cda29d84680ac253bc2de2a2cfe [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:36:08.229720 21114 raft_consensus.cc:2454] T 8ca0c1466fab4ff0927822abe5ce810a P 117eb9b3303241de8c51cfbedb5c228e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6522d4891b349c68e1bb1a298a8ddeb in term 1.
I20260619 14:36:08.229720 20983 raft_consensus.cc:2454] T 8ca0c1466fab4ff0927822abe5ce810a P 574d7cda29d84680ac253bc2de2a2cfe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6522d4891b349c68e1bb1a298a8ddeb in term 1.
I20260619 14:36:08.230038 20786 leader_election.cc:302] T 8ca0c1466fab4ff0927822abe5ce810a P e6522d4891b349c68e1bb1a298a8ddeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 117eb9b3303241de8c51cfbedb5c228e, e6522d4891b349c68e1bb1a298a8ddeb; no voters:
I20260619 14:36:08.230263 21183 raft_consensus.cc:2781] T 8ca0c1466fab4ff0927822abe5ce810a P e6522d4891b349c68e1bb1a298a8ddeb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:36:08.230525 21183 raft_consensus.cc:686] T 8ca0c1466fab4ff0927822abe5ce810a P e6522d4891b349c68e1bb1a298a8ddeb [term 1 LEADER]: Becoming Leader. State: Replica: e6522d4891b349c68e1bb1a298a8ddeb, State: Running, Role: LEADER
I20260619 14:36:08.230903 21183 consensus_queue.cc:237] T 8ca0c1466fab4ff0927822abe5ce810a P e6522d4891b349c68e1bb1a298a8ddeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6522d4891b349c68e1bb1a298a8ddeb" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 33773 } } peers { permanent_uuid: "574d7cda29d84680ac253bc2de2a2cfe" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 42003 } } peers { permanent_uuid: "117eb9b3303241de8c51cfbedb5c228e" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 33039 } }
I20260619 14:36:08.234055 20710 catalog_manager.cc:5495] T 8ca0c1466fab4ff0927822abe5ce810a P e6522d4891b349c68e1bb1a298a8ddeb reported cstate change: term changed from 0 to 1, leader changed from <none> to e6522d4891b349c68e1bb1a298a8ddeb (127.2.2.1). New cstate: current_term: 1 leader_uuid: "e6522d4891b349c68e1bb1a298a8ddeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6522d4891b349c68e1bb1a298a8ddeb" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 33773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "574d7cda29d84680ac253bc2de2a2cfe" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 42003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "117eb9b3303241de8c51cfbedb5c228e" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 33039 } health_report { overall_health: UNKNOWN } } }
I20260619 14:36:08.252130 21173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:08.248758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60870 (local address 127.2.2.1:33773)
0619 14:36:08.248898 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:08.248902 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:08.248914 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:36:08.248982 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:08.248984 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:08.249039 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:08.249129 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:36:08.249354 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:08.249845 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:08.250725 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:08.250940 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:08.251021 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:08.251296 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:36:08.251700 (+ 404us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:36:08.251813 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:08.251914 (+ 101us) server_negotiation.cc:299] Negotiation successful
0619 14:36:08.251973 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 14:36:08.258329 20983 raft_consensus.cc:1261] T 8ca0c1466fab4ff0927822abe5ce810a P 574d7cda29d84680ac253bc2de2a2cfe [term 1 FOLLOWER]: Refusing update from remote peer e6522d4891b349c68e1bb1a298a8ddeb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:36:08.258329 21114 raft_consensus.cc:1261] T 8ca0c1466fab4ff0927822abe5ce810a P 117eb9b3303241de8c51cfbedb5c228e [term 1 FOLLOWER]: Refusing update from remote peer e6522d4891b349c68e1bb1a298a8ddeb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:36:08.259143 21183 consensus_queue.cc:1035] T 8ca0c1466fab4ff0927822abe5ce810a P e6522d4891b349c68e1bb1a298a8ddeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "117eb9b3303241de8c51cfbedb5c228e" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 33039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:36:08.259346 21188 consensus_queue.cc:1035] T 8ca0c1466fab4ff0927822abe5ce810a P e6522d4891b349c68e1bb1a298a8ddeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "574d7cda29d84680ac253bc2de2a2cfe" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 42003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:36:08.267309 21191 mvcc.cc:205] Tried to move back new op lower bound from 7298579530778062848 to 7298579530676449280. Current Snapshot: MvccSnapshot[applied={T|T < 7298579530778062848}]
I20260619 14:36:08.269016 21192 mvcc.cc:205] Tried to move back new op lower bound from 7298579530778062848 to 7298579530676449280. Current Snapshot: MvccSnapshot[applied={T|T < 7298579530778062848 or (T in {7298579530778062848})}]
I20260619 14:36:08.276191 20708 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58298:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:36:08.276346 20708 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58298:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:36:08.278824 20708 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b97068f8dc8441aaa1e1ffeaf8609b00: Sending DeleteTablet for 3 replicas of tablet 8ca0c1466fab4ff0927822abe5ce810a
I20260619 14:36:08.279713 20829 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ca0c1466fab4ff0927822abe5ce810a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:36:08 UTC) from {username='slave'} at 127.0.0.1:60860
I20260619 14:36:08.279729 20963 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ca0c1466fab4ff0927822abe5ce810a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:36:08 UTC) from {username='slave'} at 127.0.0.1:35146
I20260619 14:36:08.280254 21200 tablet_replica.cc:330] T 8ca0c1466fab4ff0927822abe5ce810a P e6522d4891b349c68e1bb1a298a8ddeb: stopping tablet replica
I20260619 14:36:08.280457 21094 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ca0c1466fab4ff0927822abe5ce810a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:36:08 UTC) from {username='slave'} at 127.0.0.1:59412
I20260619 14:36:08.280716 21200 raft_consensus.cc:2229] T 8ca0c1466fab4ff0927822abe5ce810a P e6522d4891b349c68e1bb1a298a8ddeb [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:36:08.281019 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 20770
I20260619 14:36:08.281742 21201 tablet_replica.cc:330] T 8ca0c1466fab4ff0927822abe5ce810a P 574d7cda29d84680ac253bc2de2a2cfe: stopping tablet replica
I20260619 14:36:08.282188 21201 raft_consensus.cc:2229] T 8ca0c1466fab4ff0927822abe5ce810a P 574d7cda29d84680ac253bc2de2a2cfe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:36:08.282533 21201 raft_consensus.cc:2258] T 8ca0c1466fab4ff0927822abe5ce810a P 574d7cda29d84680ac253bc2de2a2cfe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:36:08.283160 21202 tablet_replica.cc:330] T 8ca0c1466fab4ff0927822abe5ce810a P 117eb9b3303241de8c51cfbedb5c228e: stopping tablet replica
I20260619 14:36:08.283438 21202 raft_consensus.cc:2229] T 8ca0c1466fab4ff0927822abe5ce810a P 117eb9b3303241de8c51cfbedb5c228e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:36:08.283730 21202 raft_consensus.cc:2258] T 8ca0c1466fab4ff0927822abe5ce810a P 117eb9b3303241de8c51cfbedb5c228e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:36:08.284271 21201 ts_tablet_manager.cc:1905] T 8ca0c1466fab4ff0927822abe5ce810a P 574d7cda29d84680ac253bc2de2a2cfe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:36:08.285003 21202 ts_tablet_manager.cc:1905] T 8ca0c1466fab4ff0927822abe5ce810a P 117eb9b3303241de8c51cfbedb5c228e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:36:08.287336 21202 ts_tablet_manager.cc:1918] T 8ca0c1466fab4ff0927822abe5ce810a P 117eb9b3303241de8c51cfbedb5c228e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:36:08.287453 21202 log.cc:1196] T 8ca0c1466fab4ff0927822abe5ce810a P 117eb9b3303241de8c51cfbedb5c228e: Deleting WAL directory at /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/ts-2/wal/wals/8ca0c1466fab4ff0927822abe5ce810a
I20260619 14:36:08.287729 21202 ts_tablet_manager.cc:1939] T 8ca0c1466fab4ff0927822abe5ce810a P 117eb9b3303241de8c51cfbedb5c228e: Deleting consensus metadata
I20260619 14:36:08.287871 21201 ts_tablet_manager.cc:1918] T 8ca0c1466fab4ff0927822abe5ce810a P 574d7cda29d84680ac253bc2de2a2cfe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:36:08.287979 21201 log.cc:1196] T 8ca0c1466fab4ff0927822abe5ce810a P 574d7cda29d84680ac253bc2de2a2cfe: Deleting WAL directory at /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879514993456-2056-0/minicluster-data/ts-1/wal/wals/8ca0c1466fab4ff0927822abe5ce810a
I20260619 14:36:08.288308 21201 ts_tablet_manager.cc:1939] T 8ca0c1466fab4ff0927822abe5ce810a P 574d7cda29d84680ac253bc2de2a2cfe: Deleting consensus metadata
W20260619 14:36:08.289065 20696 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.2.1:33773 (error 108)
I20260619 14:36:08.289126 20695 catalog_manager.cc:4841] TS 117eb9b3303241de8c51cfbedb5c228e (127.2.2.3:33039): tablet 8ca0c1466fab4ff0927822abe5ce810a (table test-table [id=254fc5ecf6cc498c90929c15652dcba5]) successfully deleted
I20260619 14:36:08.289505 20695 catalog_manager.cc:4841] TS 574d7cda29d84680ac253bc2de2a2cfe (127.2.2.2:42003): tablet 8ca0c1466fab4ff0927822abe5ce810a (table test-table [id=254fc5ecf6cc498c90929c15652dcba5]) successfully deleted
I20260619 14:36:08.289682 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 20901
I20260619 14:36:08.294703 21171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:08.294387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.1:33773 (local address 127.0.0.1:60874)
0619 14:36:08.294529 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:08.294618 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.1:33773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260619 14:36:08.294895 20696 catalog_manager.cc:4584] TS e6522d4891b349c68e1bb1a298a8ddeb (127.2.2.1:33773): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8ca0c1466fab4ff0927822abe5ce810a: Network error: Client connection negotiation failed: client connection to 127.2.2.1:33773: connect: Connection refused (error 111)
I20260619 14:36:08.295651 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 21032
I20260619 14:36:08.301447 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 20679
2026-06-19T14:36:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1194 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:36:08 dist-test-slave-70rm krb5kdc[21206](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:36:08 dist-test-slave-70rm krb5kdc[21206](info): set up 2 sockets
Jun 19 14:36:08 dist-test-slave-70rm krb5kdc[21206](info): commencing operation
krb5kdc: starting...
W20260619 14:36:09.334056 2056 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Jun 19 14:36:09 dist-test-slave-70rm krb5kdc[21206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:36:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:36:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.2.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab.
Entry for principal kudu/127.2.2.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab.
I20260619 14:36:09.458751 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:41225
--webserver_interface=127.2.2.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:38261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.2.62:41225
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:36:09.571278 21219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:36:09.571568 21219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:36:09.571610 21219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:36:09.575707 21219 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:36:09.575769 21219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:36:09.575789 21219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:36:09.575805 21219 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:36:09.575875 21219 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:36:09.582221 21219 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:38261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.2.62:41225
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:41225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.2.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:36:09.583787 21219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:36:09.585258 21219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:36:09.594769 21224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:09.594786 21225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:09.594805 21227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:09.595046 21219 server_base.cc:1034] running on GCE node
I20260619 14:36:09.595677 21219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:36:09.596817 21219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:36:09.597980 21219 hybrid_clock.cc:648] HybridClock initialized: now 1781879769597950 us; error 46 us; skew 500 ppm
Jun 19 14:36:09 dist-test-slave-70rm krb5kdc[21206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879769, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:36:09.600741 21219 init.cc:377] Logged in from keytab as kudu/127.2.2.62@KRBTEST.COM (short username kudu)
I20260619 14:36:09.601709 21219 webserver.cc:460] Webserver started at http://127.2.2.62:45139/ using document root <none> and password file <none>
I20260619 14:36:09.602355 21219 fs_manager.cc:359] Metadata directory not provided
I20260619 14:36:09.602447 21219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:36:09.602684 21219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:36:09.604754 21219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/master-0/data/instance:
uuid: "4c2e598d45824b608635586ee8618e69"
format_stamp: "Formatted at 2026-06-19 14:36:09 on dist-test-slave-70rm"
server_key: "bb064e50fdc4451c0882ca2a963d8d7d"
server_key_iv: "45ef7c13a61be69ec238ce045d4fe7eb"
server_key_version: "encryptionkey@0"
I20260619 14:36:09.605347 21219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/master-0/wal/instance:
uuid: "4c2e598d45824b608635586ee8618e69"
format_stamp: "Formatted at 2026-06-19 14:36:09 on dist-test-slave-70rm"
server_key: "bb064e50fdc4451c0882ca2a963d8d7d"
server_key_iv: "45ef7c13a61be69ec238ce045d4fe7eb"
server_key_version: "encryptionkey@0"
I20260619 14:36:09.609196 21219 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 14:36:09.611827 21234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:09.613116 21219 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:36:09.613279 21219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/master-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/master-0/wal
uuid: "4c2e598d45824b608635586ee8618e69"
format_stamp: "Formatted at 2026-06-19 14:36:09 on dist-test-slave-70rm"
server_key: "bb064e50fdc4451c0882ca2a963d8d7d"
server_key_iv: "45ef7c13a61be69ec238ce045d4fe7eb"
server_key_version: "encryptionkey@0"
I20260619 14:36:09.613410 21219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:36:09.642073 21219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:36:09.645234 21219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:36:09.645457 21219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:36:09.652733 21219 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.62:41225
I20260619 14:36:09.652760 21286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.62:41225 every 8 connection(s)
I20260619 14:36:09.653796 21219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
I20260619 14:36:09.655146 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 21219
I20260619 14:36:09.655303 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/master-0/wal/instance
I20260619 14:36:09.655575 2056 external_mini_cluster.cc:1428] Setting key 912c647ad7ee6f3622a8e000bc17a757
I20260619 14:36:09.657006 21287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:36:09.666038 21287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c2e598d45824b608635586ee8618e69: Bootstrap starting.
Jun 19 14:36:09 dist-test-slave-70rm krb5kdc[21206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:36:09.668953 21287 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c2e598d45824b608635586ee8618e69: Neither blocks nor log segments found. Creating new log.
I20260619 14:36:09.669714 21287 log.cc:824] T 00000000000000000000000000000000 P 4c2e598d45824b608635586ee8618e69: Log is configured to *not* fsync() on all Append() calls
I20260619 14:36:09.671831 21287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c2e598d45824b608635586ee8618e69: No bootstrap required, opened a new log
I20260619 14:36:09.672252 21290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:09.657090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46170 (local address 127.2.2.62:41225)
0619 14:36:09.657569 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:09.657585 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:09.657631 (+ 46us) server_negotiation.cc:407] Connection header received
0619 14:36:09.658377 (+ 746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:09.658396 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:09.658773 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:09.659135 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:09.659990 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:09.662526 (+ 2536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:09.663488 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:09.664011 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:09.664197 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:09.667185 (+ 2988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:09.667191 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:09.667206 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:36:09.667250 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:09.669047 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:09.669537 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:09.669550 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:09.669556 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:09.669672 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:09.669992 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:09.670000 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:09.670001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:09.670604 (+ 603us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:09.670806 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:09.671131 (+ 325us) server_negotiation.cc:299] Negotiation successful
0619 14:36:09.671447 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":168,"threads_started":1}
I20260619 14:36:09.676029 21287 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c2e598d45824b608635586ee8618e69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c2e598d45824b608635586ee8618e69" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41225 } }
I20260619 14:36:09.676291 21287 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c2e598d45824b608635586ee8618e69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:36:09.676373 21287 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c2e598d45824b608635586ee8618e69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c2e598d45824b608635586ee8618e69, State: Initialized, Role: FOLLOWER
I20260619 14:36:09.676849 21287 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c2e598d45824b608635586ee8618e69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c2e598d45824b608635586ee8618e69" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41225 } }
I20260619 14:36:09.677018 21287 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c2e598d45824b608635586ee8618e69 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:36:09.677115 21287 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c2e598d45824b608635586ee8618e69 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:36:09.677518 21287 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c2e598d45824b608635586ee8618e69 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:36:09.678908 21287 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c2e598d45824b608635586ee8618e69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c2e598d45824b608635586ee8618e69" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41225 } }
I20260619 14:36:09.679302 21287 leader_election.cc:302] T 00000000000000000000000000000000 P 4c2e598d45824b608635586ee8618e69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c2e598d45824b608635586ee8618e69; no voters:
I20260619 14:36:09.679690 21287 leader_election.cc:288] T 00000000000000000000000000000000 P 4c2e598d45824b608635586ee8618e69 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:36:09.679865 21292 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c2e598d45824b608635586ee8618e69 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:36:09.680114 21292 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c2e598d45824b608635586ee8618e69 [term 1 LEADER]: Becoming Leader. State: Replica: 4c2e598d45824b608635586ee8618e69, State: Running, Role: LEADER
I20260619 14:36:09.680536 21292 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c2e598d45824b608635586ee8618e69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c2e598d45824b608635586ee8618e69" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41225 } }
I20260619 14:36:09.681110 21287 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c2e598d45824b608635586ee8618e69 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:36:09.682413 21293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c2e598d45824b608635586ee8618e69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c2e598d45824b608635586ee8618e69. Latest consensus state: current_term: 1 leader_uuid: "4c2e598d45824b608635586ee8618e69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c2e598d45824b608635586ee8618e69" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41225 } } }
I20260619 14:36:09.682549 21293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c2e598d45824b608635586ee8618e69 [sys.catalog]: This master's current role is: LEADER
I20260619 14:36:09.682863 21294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c2e598d45824b608635586ee8618e69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c2e598d45824b608635586ee8618e69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c2e598d45824b608635586ee8618e69" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41225 } } }
I20260619 14:36:09.682945 21294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c2e598d45824b608635586ee8618e69 [sys.catalog]: This master's current role is: LEADER
I20260619 14:36:09.682996 21301 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:36:09.685822 21301 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:36:09.691854 21301 catalog_manager.cc:1347] Generated new cluster ID: 47651bcce4f84e79853e4b706587ca7a
I20260619 14:36:09.691941 21301 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:36:09.711517 21301 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:36:09.712845 21301 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:36:09.719043 21301 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c2e598d45824b608635586ee8618e69: Generated new TSK 0
I20260619 14:36:09.719960 21301 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.2.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab.
Entry for principal kudu/127.2.2.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab.
I20260619 14:36:09.757743 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.1:0
--local_ip_for_outbound_sockets=127.2.2.1
--webserver_interface=127.2.2.1
--webserver_port=0
--tserver_master_addrs=127.2.2.62:41225
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab
--principal=kudu/127.2.2.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:38261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:36:09.882216 21313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:36:09.882507 21313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:36:09.882552 21313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:36:09.886794 21313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:36:09.886865 21313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:36:09.886927 21313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.1
I20260619 14:36:09.892406 21313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:38261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.1.keytab
--principal=kudu/127.2.2.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.2.1
--webserver_port=0
--tserver_master_addrs=127.2.2.62:41225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.1
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:36:09.893457 21313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:36:09.894544 21313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:36:09.904244 21313 server_base.cc:1034] running on GCE node
W20260619 14:36:09.904184 21319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:09.904201 21318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:09.904189 21321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:09.904897 21313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:36:09.905577 21313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:36:09.906764 21313 hybrid_clock.cc:648] HybridClock initialized: now 1781879769906750 us; error 33 us; skew 500 ppm
Jun 19 14:36:09 dist-test-slave-70rm krb5kdc[21206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879769, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:36:09.909391 21313 init.cc:377] Logged in from keytab as kudu/127.2.2.1@KRBTEST.COM (short username kudu)
I20260619 14:36:09.910362 21313 webserver.cc:460] Webserver started at http://127.2.2.1:33095/ using document root <none> and password file <none>
I20260619 14:36:09.910934 21313 fs_manager.cc:359] Metadata directory not provided
I20260619 14:36:09.911026 21313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:36:09.911265 21313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:36:09.913362 21313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/ts-0/data/instance:
uuid: "3ca2ce1807214f85b25ca04d9e8e2918"
format_stamp: "Formatted at 2026-06-19 14:36:09 on dist-test-slave-70rm"
server_key: "b719910ad67c9942229c96097ac077ad"
server_key_iv: "354da31145c7d2d4f2361590ac13f0a3"
server_key_version: "encryptionkey@0"
I20260619 14:36:09.913949 21313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/ts-0/wal/instance:
uuid: "3ca2ce1807214f85b25ca04d9e8e2918"
format_stamp: "Formatted at 2026-06-19 14:36:09 on dist-test-slave-70rm"
server_key: "b719910ad67c9942229c96097ac077ad"
server_key_iv: "354da31145c7d2d4f2361590ac13f0a3"
server_key_version: "encryptionkey@0"
I20260619 14:36:09.917861 21313 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260619 14:36:09.920418 21328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:09.921649 21313 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:36:09.921798 21313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/ts-0/wal
uuid: "3ca2ce1807214f85b25ca04d9e8e2918"
format_stamp: "Formatted at 2026-06-19 14:36:09 on dist-test-slave-70rm"
server_key: "b719910ad67c9942229c96097ac077ad"
server_key_iv: "354da31145c7d2d4f2361590ac13f0a3"
server_key_version: "encryptionkey@0"
I20260619 14:36:09.921917 21313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:36:09.952848 21313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:36:09.955957 21313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:36:09.956195 21313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:36:09.956827 21313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:36:09.957710 21313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:36:09.957783 21313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:09.957856 21313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:36:09.957912 21313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:09.967475 21313 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.1:36067
I20260619 14:36:09.967497 21441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.1:36067 every 8 connection(s)
I20260619 14:36:09.968421 21313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
I20260619 14:36:09.975203 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 21313
I20260619 14:36:09.975354 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/ts-0/wal/instance
I20260619 14:36:09.975688 2056 external_mini_cluster.cc:1428] Setting key 9d33bb20fc56b36808b6bc2350ea5d87
Jun 19 14:36:09 dist-test-slave-70rm krb5kdc[21206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879769, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.1@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:36:09.983426 21290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:09.970347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:49465 (local address 127.2.2.62:41225)
0619 14:36:09.970492 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:09.970495 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:09.971422 (+ 927us) server_negotiation.cc:407] Connection header received
0619 14:36:09.972678 (+ 1256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:09.972682 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:09.972731 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:09.972809 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:09.974069 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:09.974707 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:09.976387 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:09.976621 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:09.976753 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:09.979505 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:09.979510 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:09.979512 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:36:09.979548 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:09.981073 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:09.981618 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:09.981621 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:09.981623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:09.981741 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:09.982216 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:09.982226 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:09.982228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:09.982428 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:09.982535 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:09.983102 (+ 567us) server_negotiation.cc:299] Negotiation successful
0619 14:36:09.983249 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 14:36:09.984280 21444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:09.970796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41225 (local address 127.2.2.1:49465)
0619 14:36:09.971266 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:09.971311 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 14:36:09.972467 (+ 1156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:36:09.972978 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:36:09.972985 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:36:09.973365 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:36:09.973845 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:09.973858 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:09.974843 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:09.974846 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:09.976263 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:09.976276 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:09.976732 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:09.976796 (+ 64us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:09.977039 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:09.977735 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:36:09.977753 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:36:09.979317 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:36:09.981203 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:36:09.981209 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:36:09.981228 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:36:09.981507 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:36:09.981842 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:36:09.981846 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:36:09.981849 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:36:09.982087 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:36:09.982534 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:36:09.982542 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:36:09.982834 (+ 292us) client_negotiation.cc:769] Sending connection context
0619 14:36:09.983046 (+ 212us) client_negotiation.cc:240] Negotiation successful
0619 14:36:09.983401 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":123,"threads_started":1}
WARNING: no policy specified for kudu/127.2.2.2@KRBTEST.COM; defaulting to no policy
I20260619 14:36:09.985671 21442 heartbeater.cc:347] Connected to a master server at 127.2.2.62:41225
I20260619 14:36:09.985934 21442 heartbeater.cc:464] Registering TS with master...
I20260619 14:36:09.986497 21442 heartbeater.cc:511] Master 127.2.2.62:41225 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.2@KRBTEST.COM" created.
I20260619 14:36:09.988109 21251 ts_manager.cc:194] Registered new tserver with Master: 3ca2ce1807214f85b25ca04d9e8e2918 (127.2.2.1:36067)
I20260619 14:36:09.989300 21251 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.2.1@KRBTEST.COM'} at 127.2.2.1:49465
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab.
Entry for principal kudu/127.2.2.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab.
I20260619 14:36:10.003342 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.2:0
--local_ip_for_outbound_sockets=127.2.2.2
--webserver_interface=127.2.2.2
--webserver_port=0
--tserver_master_addrs=127.2.2.62:41225
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab
--principal=kudu/127.2.2.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:38261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:36:10.115296 21447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:36:10.115592 21447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:36:10.115638 21447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:36:10.119833 21447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:36:10.119905 21447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:36:10.119968 21447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.2
I20260619 14:36:10.125527 21447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:38261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.2.keytab
--principal=kudu/127.2.2.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.2.2
--webserver_port=0
--tserver_master_addrs=127.2.2.62:41225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.2
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:36:10.126741 21447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:36:10.127766 21447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:36:10.137537 21453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:10.137684 21447 server_base.cc:1034] running on GCE node
W20260619 14:36:10.137573 21452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:10.137887 21455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:10.138347 21447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:36:10.139065 21447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:36:10.140256 21447 hybrid_clock.cc:648] HybridClock initialized: now 1781879770140233 us; error 35 us; skew 500 ppm
Jun 19 14:36:10 dist-test-slave-70rm krb5kdc[21206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879770, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:36:10.143302 21447 init.cc:377] Logged in from keytab as kudu/127.2.2.2@KRBTEST.COM (short username kudu)
I20260619 14:36:10.144194 21447 webserver.cc:460] Webserver started at http://127.2.2.2:46041/ using document root <none> and password file <none>
I20260619 14:36:10.144773 21447 fs_manager.cc:359] Metadata directory not provided
I20260619 14:36:10.144865 21447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:36:10.145095 21447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:36:10.147203 21447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/ts-1/data/instance:
uuid: "97f600aa840d4439a6e1c6a2d541ebe4"
format_stamp: "Formatted at 2026-06-19 14:36:10 on dist-test-slave-70rm"
server_key: "dc3aa27218e79aa5bc4b8f341ef983ff"
server_key_iv: "b5e9b05d7790c24a7bd7da55f77d8af3"
server_key_version: "encryptionkey@0"
I20260619 14:36:10.147768 21447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/ts-1/wal/instance:
uuid: "97f600aa840d4439a6e1c6a2d541ebe4"
format_stamp: "Formatted at 2026-06-19 14:36:10 on dist-test-slave-70rm"
server_key: "dc3aa27218e79aa5bc4b8f341ef983ff"
server_key_iv: "b5e9b05d7790c24a7bd7da55f77d8af3"
server_key_version: "encryptionkey@0"
I20260619 14:36:10.151588 21447 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:36:10.153980 21462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:10.155231 21447 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:36:10.155365 21447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/ts-1/wal
uuid: "97f600aa840d4439a6e1c6a2d541ebe4"
format_stamp: "Formatted at 2026-06-19 14:36:10 on dist-test-slave-70rm"
server_key: "dc3aa27218e79aa5bc4b8f341ef983ff"
server_key_iv: "b5e9b05d7790c24a7bd7da55f77d8af3"
server_key_version: "encryptionkey@0"
I20260619 14:36:10.155481 21447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:36:10.178236 21447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:36:10.182030 21447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:36:10.182277 21447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:36:10.183010 21447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:36:10.183924 21447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:36:10.183993 21447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:10.184065 21447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:36:10.184118 21447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:10.193193 21447 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.2:43217
I20260619 14:36:10.193214 21575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.2:43217 every 8 connection(s)
I20260619 14:36:10.194242 21447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
I20260619 14:36:10.199975 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 21447
I20260619 14:36:10.200105 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/ts-1/wal/instance
I20260619 14:36:10.200352 2056 external_mini_cluster.cc:1428] Setting key f610885832cdb08f9661a51e34d3a9d5
Jun 19 14:36:10 dist-test-slave-70rm krb5kdc[21206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879770, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.2@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:36:10.208143 21290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:10.196040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:34467 (local address 127.2.2.62:41225)
0619 14:36:10.196173 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:10.196177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:10.196960 (+ 783us) server_negotiation.cc:407] Connection header received
0619 14:36:10.198336 (+ 1376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:10.198339 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:10.198397 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:10.198477 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:10.199645 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:10.200305 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:10.201468 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:10.201691 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:10.201809 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:10.204471 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:10.204474 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:10.204476 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:36:10.204499 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:10.205997 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:10.206571 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:10.206574 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:10.206576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:10.206660 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:10.207009 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:10.207013 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:10.207015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:10.207207 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:10.207320 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:10.207843 (+ 523us) server_negotiation.cc:299] Negotiation successful
0619 14:36:10.207970 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 14:36:10.208985 21578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:10.196388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41225 (local address 127.2.2.2:34467)
0619 14:36:10.196812 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:10.196855 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 14:36:10.198058 (+ 1203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:36:10.198648 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:36:10.198655 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:36:10.199009 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:36:10.199495 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:10.199508 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:10.200438 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:10.200442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:10.201326 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:10.201336 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:10.201796 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:10.201850 (+ 54us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:10.202020 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:10.202722 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:36:10.202740 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:36:10.204284 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:36:10.206167 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:36:10.206173 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:36:10.206196 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:36:10.206459 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:36:10.206759 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:36:10.206763 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:36:10.206765 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:36:10.206907 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:36:10.207327 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:36:10.207335 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:36:10.207591 (+ 256us) client_negotiation.cc:769] Sending connection context
0619 14:36:10.207787 (+ 196us) client_negotiation.cc:240] Negotiation successful
0619 14:36:10.208119 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":111,"threads_started":1}
WARNING: no policy specified for kudu/127.2.2.3@KRBTEST.COM; defaulting to no policy
I20260619 14:36:10.210172 21576 heartbeater.cc:347] Connected to a master server at 127.2.2.62:41225
I20260619 14:36:10.210368 21576 heartbeater.cc:464] Registering TS with master...
I20260619 14:36:10.210845 21576 heartbeater.cc:511] Master 127.2.2.62:41225 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.3@KRBTEST.COM" created.
I20260619 14:36:10.211956 21251 ts_manager.cc:194] Registered new tserver with Master: 97f600aa840d4439a6e1c6a2d541ebe4 (127.2.2.2:43217)
I20260619 14:36:10.212800 21251 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.2.2@KRBTEST.COM'} at 127.2.2.2:34467
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab.
Entry for principal kudu/127.2.2.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab.
I20260619 14:36:10.226980 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.3:0
--local_ip_for_outbound_sockets=127.2.2.3
--webserver_interface=127.2.2.3
--webserver_port=0
--tserver_master_addrs=127.2.2.62:41225
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab
--principal=kudu/127.2.2.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:38261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:36:10.338279 21581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:36:10.338575 21581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:36:10.338621 21581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:36:10.342888 21581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:36:10.342957 21581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:36:10.343065 21581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.3
I20260619 14:36:10.348560 21581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:38261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.3.keytab
--principal=kudu/127.2.2.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.2.3
--webserver_port=0
--tserver_master_addrs=127.2.2.62:41225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.3
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:36:10.349738 21581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:36:10.350868 21581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:36:10.361130 21586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:10.361146 21589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:10.361159 21587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:10.361210 21581 server_base.cc:1034] running on GCE node
I20260619 14:36:10.361975 21581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:36:10.362782 21581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:36:10.363986 21581 hybrid_clock.cc:648] HybridClock initialized: now 1781879770363957 us; error 39 us; skew 500 ppm
Jun 19 14:36:10 dist-test-slave-70rm krb5kdc[21206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879770, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:36:10.366927 21581 init.cc:377] Logged in from keytab as kudu/127.2.2.3@KRBTEST.COM (short username kudu)
I20260619 14:36:10.367866 21581 webserver.cc:460] Webserver started at http://127.2.2.3:42913/ using document root <none> and password file <none>
I20260619 14:36:10.368433 21581 fs_manager.cc:359] Metadata directory not provided
I20260619 14:36:10.368522 21581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:36:10.368757 21581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:36:10.370865 21581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/ts-2/data/instance:
uuid: "504012ca24084b3ba039993e0ae2fa89"
format_stamp: "Formatted at 2026-06-19 14:36:10 on dist-test-slave-70rm"
server_key: "4b995e9919a3f2dcdfa56c7e49bf4e51"
server_key_iv: "c2833c576b03d480afd31f6cf116fa7a"
server_key_version: "encryptionkey@0"
I20260619 14:36:10.371431 21581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/ts-2/wal/instance:
uuid: "504012ca24084b3ba039993e0ae2fa89"
format_stamp: "Formatted at 2026-06-19 14:36:10 on dist-test-slave-70rm"
server_key: "4b995e9919a3f2dcdfa56c7e49bf4e51"
server_key_iv: "c2833c576b03d480afd31f6cf116fa7a"
server_key_version: "encryptionkey@0"
I20260619 14:36:10.375397 21581 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 14:36:10.378561 21596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:10.379815 21581 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20260619 14:36:10.379916 21581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/ts-2/wal
uuid: "504012ca24084b3ba039993e0ae2fa89"
format_stamp: "Formatted at 2026-06-19 14:36:10 on dist-test-slave-70rm"
server_key: "4b995e9919a3f2dcdfa56c7e49bf4e51"
server_key_iv: "c2833c576b03d480afd31f6cf116fa7a"
server_key_version: "encryptionkey@0"
I20260619 14:36:10.380009 21581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:36:10.400264 21581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:36:10.403818 21581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:36:10.403981 21581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:36:10.404523 21581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:36:10.405309 21581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:36:10.405357 21581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:10.405398 21581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:36:10.405421 21581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:10.414618 21581 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.3:45643
I20260619 14:36:10.414633 21709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.3:45643 every 8 connection(s)
I20260619 14:36:10.415565 21581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
I20260619 14:36:10.423796 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 21581
I20260619 14:36:10.423949 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/ts-2/wal/instance
I20260619 14:36:10.424284 2056 external_mini_cluster.cc:1428] Setting key 61b374b33389d8f6f58f46546395647b
Jun 19 14:36:10 dist-test-slave-70rm krb5kdc[21206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879770, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.3@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:36:10.429802 21290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:10.417450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.3:33439 (local address 127.2.2.62:41225)
0619 14:36:10.417612 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:10.417617 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:10.418631 (+ 1014us) server_negotiation.cc:407] Connection header received
0619 14:36:10.420005 (+ 1374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:10.420007 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:10.420055 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:10.420126 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:10.421303 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:10.421814 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:10.423158 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:10.423385 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:10.423467 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:10.426158 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:10.426162 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:10.426165 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:36:10.426196 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:10.427621 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:10.428198 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:10.428202 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:10.428204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:10.428293 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:10.428651 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:10.428655 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:10.428657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:10.428861 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:10.428972 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:10.429494 (+ 522us) server_negotiation.cc:299] Negotiation successful
0619 14:36:10.429637 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 14:36:10.430725 21712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:10.417950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41225 (local address 127.2.2.3:33439)
0619 14:36:10.418473 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:10.418526 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 14:36:10.419768 (+ 1242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:36:10.420273 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:36:10.420279 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:36:10.420663 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:36:10.421163 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:10.421177 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:10.421923 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:10.421926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:10.423043 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:10.423052 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:10.423470 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:10.423543 (+ 73us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:10.423753 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:10.424374 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:36:10.424394 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:36:10.425961 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:36:10.427765 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:36:10.427772 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:36:10.427788 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:36:10.428075 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:36:10.428396 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:36:10.428399 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:36:10.428401 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:36:10.428549 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:36:10.428980 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:36:10.428990 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:36:10.429252 (+ 262us) client_negotiation.cc:769] Sending connection context
0619 14:36:10.429454 (+ 202us) client_negotiation.cc:240] Negotiation successful
0619 14:36:10.429800 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":192,"threads_started":1}
I20260619 14:36:10.431874 21710 heartbeater.cc:347] Connected to a master server at 127.2.2.62:41225
I20260619 14:36:10.432106 21710 heartbeater.cc:464] Registering TS with master...
I20260619 14:36:10.432562 21710 heartbeater.cc:511] Master 127.2.2.62:41225 requested a full tablet report, sending...
I20260619 14:36:10.433655 21251 ts_manager.cc:194] Registered new tserver with Master: 504012ca24084b3ba039993e0ae2fa89 (127.2.2.3:45643)
I20260619 14:36:10.434587 21251 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.2.3@KRBTEST.COM'} at 127.2.2.3:33439
I20260619 14:36:10.438082 2056 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:36:10.446918 21290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:10.439981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46176 (local address 127.2.2.62:41225)
0619 14:36:10.440117 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:10.440120 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:10.440242 (+ 122us) server_negotiation.cc:407] Connection header received
0619 14:36:10.440355 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:10.440357 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:10.440394 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:10.440451 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:10.441121 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:10.441615 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:10.442483 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:10.442689 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:10.442775 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:10.444047 (+ 1272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:10.444050 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:10.444053 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:36:10.444075 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:10.445526 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:10.445882 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:10.445884 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:10.445885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:10.445954 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:10.446217 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:10.446220 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:10.446221 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:10.446427 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:10.446531 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:10.446701 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 14:36:10.446799 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 14:36:10.449807 21251 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46176:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:36:10.452993 21251 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:36:10.467537 21726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:10.462110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52604 (local address 127.2.2.2:43217)
0619 14:36:10.462674 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:10.462686 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:10.462708 (+ 22us) server_negotiation.cc:407] Connection header received
0619 14:36:10.462783 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:10.462809 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:10.463032 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:10.463186 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:36:10.463447 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:10.464558 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:10.466071 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:10.466811 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:10.466938 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:10.467086 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:10.467261 (+ 175us) server_negotiation.cc:299] Negotiation successful
0619 14:36:10.467357 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":100,"threads_started":1}
I20260619 14:36:10.467552 21724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:10.461400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.2:43217 (local address 127.0.0.1:52604)
0619 14:36:10.462159 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:10.462174 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:36:10.462246 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:36:10.463176 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:36:10.463182 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:36:10.463201 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:36:10.463326 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:10.463333 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:10.464685 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:10.464689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:10.465945 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:10.465956 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:10.466938 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:10.466941 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:10.467052 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:10.467087 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 14:36:10.467239 (+ 152us) client_negotiation.cc:240] Negotiation successful
0619 14:36:10.467360 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"mutex_wait_us":55,"thread_start_us":90,"threads_started":1}
I20260619 14:36:10.467597 21721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:10.461329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.1:36067 (local address 127.0.0.1:60274)
0619 14:36:10.461928 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:10.461978 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 14:36:10.462108 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:36:10.463016 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:36:10.463022 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:36:10.463045 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:36:10.463201 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:10.463208 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:10.464391 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:10.464394 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:10.465946 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:10.465956 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:10.467150 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:10.467154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:10.467258 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:10.467281 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 14:36:10.467357 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 14:36:10.467430 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"mutex_wait_us":90,"thread_start_us":91,"threads_started":1}
I20260619 14:36:10.468158 21722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:10.461688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.3:45643 (local address 127.0.0.1:52094)
0619 14:36:10.462536 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:10.462551 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:36:10.462635 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:36:10.463958 (+ 1323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:36:10.463961 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:36:10.463979 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:36:10.464056 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:10.464061 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:10.465242 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:10.465245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:10.466376 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:10.466386 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:10.467553 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:10.467556 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:10.467880 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:10.467903 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 14:36:10.467978 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 14:36:10.468039 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":777,"mutex_wait_us":2,"thread_start_us":88,"threads_started":1}
I20260619 14:36:10.468166 21723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:10.461459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60274 (local address 127.2.2.1:36067)
0619 14:36:10.462466 (+ 1007us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:10.462479 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:10.462523 (+ 44us) server_negotiation.cc:407] Connection header received
0619 14:36:10.462604 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:10.462609 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:10.462852 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:10.463022 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:36:10.463324 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:10.464265 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:10.466089 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:10.466850 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:10.466953 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:10.467087 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:10.467906 (+ 819us) server_negotiation.cc:299] Negotiation successful
0619 14:36:10.468012 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":916,"thread_start_us":112,"threads_started":1}
I20260619 14:36:10.468266 21725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:10.462076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52094 (local address 127.2.2.3:45643)
0619 14:36:10.463543 (+ 1467us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:10.463554 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:10.463574 (+ 20us) server_negotiation.cc:407] Connection header received
0619 14:36:10.463644 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:10.463649 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:10.463818 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:10.463959 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:36:10.464152 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:10.465083 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:10.466500 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:10.467279 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:10.467379 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:10.467497 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:10.468029 (+ 532us) server_negotiation.cc:299] Negotiation successful
0619 14:36:10.468123 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1361,"thread_start_us":147,"threads_started":1}
I20260619 14:36:10.470728 21376 tablet_service.cc:1467] Processing CreateTablet for tablet d98a21a4822e4fb2b054efe1eb6c0354 (DEFAULT_TABLE table=test-table [id=11f70167e2004a2f8e701e7d7d5f204e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:36:10.470767 21510 tablet_service.cc:1467] Processing CreateTablet for tablet d98a21a4822e4fb2b054efe1eb6c0354 (DEFAULT_TABLE table=test-table [id=11f70167e2004a2f8e701e7d7d5f204e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:36:10.470983 21644 tablet_service.cc:1467] Processing CreateTablet for tablet d98a21a4822e4fb2b054efe1eb6c0354 (DEFAULT_TABLE table=test-table [id=11f70167e2004a2f8e701e7d7d5f204e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:36:10.471655 21376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d98a21a4822e4fb2b054efe1eb6c0354. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:36:10.471782 21644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d98a21a4822e4fb2b054efe1eb6c0354. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:36:10.471782 21510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d98a21a4822e4fb2b054efe1eb6c0354. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:36:10.478083 21727 tablet_bootstrap.cc:492] T d98a21a4822e4fb2b054efe1eb6c0354 P 3ca2ce1807214f85b25ca04d9e8e2918: Bootstrap starting.
I20260619 14:36:10.479959 21728 tablet_bootstrap.cc:492] T d98a21a4822e4fb2b054efe1eb6c0354 P 504012ca24084b3ba039993e0ae2fa89: Bootstrap starting.
I20260619 14:36:10.481660 21727 tablet_bootstrap.cc:654] T d98a21a4822e4fb2b054efe1eb6c0354 P 3ca2ce1807214f85b25ca04d9e8e2918: Neither blocks nor log segments found. Creating new log.
I20260619 14:36:10.482244 21728 tablet_bootstrap.cc:654] T d98a21a4822e4fb2b054efe1eb6c0354 P 504012ca24084b3ba039993e0ae2fa89: Neither blocks nor log segments found. Creating new log.
I20260619 14:36:10.482486 21729 tablet_bootstrap.cc:492] T d98a21a4822e4fb2b054efe1eb6c0354 P 97f600aa840d4439a6e1c6a2d541ebe4: Bootstrap starting.
I20260619 14:36:10.483187 21727 log.cc:824] T d98a21a4822e4fb2b054efe1eb6c0354 P 3ca2ce1807214f85b25ca04d9e8e2918: Log is configured to *not* fsync() on all Append() calls
I20260619 14:36:10.483505 21728 log.cc:824] T d98a21a4822e4fb2b054efe1eb6c0354 P 504012ca24084b3ba039993e0ae2fa89: Log is configured to *not* fsync() on all Append() calls
I20260619 14:36:10.485140 21729 tablet_bootstrap.cc:654] T d98a21a4822e4fb2b054efe1eb6c0354 P 97f600aa840d4439a6e1c6a2d541ebe4: Neither blocks nor log segments found. Creating new log.
I20260619 14:36:10.485764 21728 tablet_bootstrap.cc:492] T d98a21a4822e4fb2b054efe1eb6c0354 P 504012ca24084b3ba039993e0ae2fa89: No bootstrap required, opened a new log
I20260619 14:36:10.485980 21728 ts_tablet_manager.cc:1397] T d98a21a4822e4fb2b054efe1eb6c0354 P 504012ca24084b3ba039993e0ae2fa89: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260619 14:36:10.485965 21729 log.cc:824] T d98a21a4822e4fb2b054efe1eb6c0354 P 97f600aa840d4439a6e1c6a2d541ebe4: Log is configured to *not* fsync() on all Append() calls
I20260619 14:36:10.486548 21727 tablet_bootstrap.cc:492] T d98a21a4822e4fb2b054efe1eb6c0354 P 3ca2ce1807214f85b25ca04d9e8e2918: No bootstrap required, opened a new log
I20260619 14:36:10.486755 21727 ts_tablet_manager.cc:1397] T d98a21a4822e4fb2b054efe1eb6c0354 P 3ca2ce1807214f85b25ca04d9e8e2918: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260619 14:36:10.488054 21729 tablet_bootstrap.cc:492] T d98a21a4822e4fb2b054efe1eb6c0354 P 97f600aa840d4439a6e1c6a2d541ebe4: No bootstrap required, opened a new log
I20260619 14:36:10.488224 21729 ts_tablet_manager.cc:1397] T d98a21a4822e4fb2b054efe1eb6c0354 P 97f600aa840d4439a6e1c6a2d541ebe4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 14:36:10.489301 21728 raft_consensus.cc:348] T d98a21a4822e4fb2b054efe1eb6c0354 P 504012ca24084b3ba039993e0ae2fa89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ca2ce1807214f85b25ca04d9e8e2918" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 36067 } } peers { permanent_uuid: "97f600aa840d4439a6e1c6a2d541ebe4" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 43217 } } peers { permanent_uuid: "504012ca24084b3ba039993e0ae2fa89" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45643 } }
I20260619 14:36:10.489593 21728 raft_consensus.cc:374] T d98a21a4822e4fb2b054efe1eb6c0354 P 504012ca24084b3ba039993e0ae2fa89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:36:10.489646 21728 raft_consensus.cc:729] T d98a21a4822e4fb2b054efe1eb6c0354 P 504012ca24084b3ba039993e0ae2fa89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 504012ca24084b3ba039993e0ae2fa89, State: Initialized, Role: FOLLOWER
I20260619 14:36:10.490177 21728 consensus_queue.cc:260] T d98a21a4822e4fb2b054efe1eb6c0354 P 504012ca24084b3ba039993e0ae2fa89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ca2ce1807214f85b25ca04d9e8e2918" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 36067 } } peers { permanent_uuid: "97f600aa840d4439a6e1c6a2d541ebe4" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 43217 } } peers { permanent_uuid: "504012ca24084b3ba039993e0ae2fa89" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45643 } }
I20260619 14:36:10.490746 21727 raft_consensus.cc:348] T d98a21a4822e4fb2b054efe1eb6c0354 P 3ca2ce1807214f85b25ca04d9e8e2918 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ca2ce1807214f85b25ca04d9e8e2918" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 36067 } } peers { permanent_uuid: "97f600aa840d4439a6e1c6a2d541ebe4" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 43217 } } peers { permanent_uuid: "504012ca24084b3ba039993e0ae2fa89" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45643 } }
I20260619 14:36:10.490980 21710 heartbeater.cc:503] Master 127.2.2.62:41225 was elected leader, sending a full tablet report...
I20260619 14:36:10.491009 21727 raft_consensus.cc:374] T d98a21a4822e4fb2b054efe1eb6c0354 P 3ca2ce1807214f85b25ca04d9e8e2918 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:36:10.491056 21727 raft_consensus.cc:729] T d98a21a4822e4fb2b054efe1eb6c0354 P 3ca2ce1807214f85b25ca04d9e8e2918 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ca2ce1807214f85b25ca04d9e8e2918, State: Initialized, Role: FOLLOWER
I20260619 14:36:10.491385 21728 ts_tablet_manager.cc:1428] T d98a21a4822e4fb2b054efe1eb6c0354 P 504012ca24084b3ba039993e0ae2fa89: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 14:36:10.491503 21727 consensus_queue.cc:260] T d98a21a4822e4fb2b054efe1eb6c0354 P 3ca2ce1807214f85b25ca04d9e8e2918 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ca2ce1807214f85b25ca04d9e8e2918" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 36067 } } peers { permanent_uuid: "97f600aa840d4439a6e1c6a2d541ebe4" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 43217 } } peers { permanent_uuid: "504012ca24084b3ba039993e0ae2fa89" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45643 } }
I20260619 14:36:10.492120 21442 heartbeater.cc:503] Master 127.2.2.62:41225 was elected leader, sending a full tablet report...
I20260619 14:36:10.491971 21729 raft_consensus.cc:348] T d98a21a4822e4fb2b054efe1eb6c0354 P 97f600aa840d4439a6e1c6a2d541ebe4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ca2ce1807214f85b25ca04d9e8e2918" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 36067 } } peers { permanent_uuid: "97f600aa840d4439a6e1c6a2d541ebe4" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 43217 } } peers { permanent_uuid: "504012ca24084b3ba039993e0ae2fa89" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45643 } }
I20260619 14:36:10.492233 21729 raft_consensus.cc:374] T d98a21a4822e4fb2b054efe1eb6c0354 P 97f600aa840d4439a6e1c6a2d541ebe4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:36:10.492293 21729 raft_consensus.cc:729] T d98a21a4822e4fb2b054efe1eb6c0354 P 97f600aa840d4439a6e1c6a2d541ebe4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97f600aa840d4439a6e1c6a2d541ebe4, State: Initialized, Role: FOLLOWER
I20260619 14:36:10.492687 21729 consensus_queue.cc:260] T d98a21a4822e4fb2b054efe1eb6c0354 P 97f600aa840d4439a6e1c6a2d541ebe4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ca2ce1807214f85b25ca04d9e8e2918" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 36067 } } peers { permanent_uuid: "97f600aa840d4439a6e1c6a2d541ebe4" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 43217 } } peers { permanent_uuid: "504012ca24084b3ba039993e0ae2fa89" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45643 } }
I20260619 14:36:10.493428 21727 ts_tablet_manager.cc:1428] T d98a21a4822e4fb2b054efe1eb6c0354 P 3ca2ce1807214f85b25ca04d9e8e2918: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260619 14:36:10.493541 21729 ts_tablet_manager.cc:1428] T d98a21a4822e4fb2b054efe1eb6c0354 P 97f600aa840d4439a6e1c6a2d541ebe4: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 14:36:10.493860 21576 heartbeater.cc:503] Master 127.2.2.62:41225 was elected leader, sending a full tablet report...
I20260619 14:36:10.528056 21735 raft_consensus.cc:482] T d98a21a4822e4fb2b054efe1eb6c0354 P 97f600aa840d4439a6e1c6a2d541ebe4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:36:10.528265 21735 raft_consensus.cc:504] T d98a21a4822e4fb2b054efe1eb6c0354 P 97f600aa840d4439a6e1c6a2d541ebe4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ca2ce1807214f85b25ca04d9e8e2918" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 36067 } } peers { permanent_uuid: "97f600aa840d4439a6e1c6a2d541ebe4" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 43217 } } peers { permanent_uuid: "504012ca24084b3ba039993e0ae2fa89" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45643 } }
I20260619 14:36:10.529618 21735 leader_election.cc:288] T d98a21a4822e4fb2b054efe1eb6c0354 P 97f600aa840d4439a6e1c6a2d541ebe4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ca2ce1807214f85b25ca04d9e8e2918 (127.2.2.1:36067), 504012ca24084b3ba039993e0ae2fa89 (127.2.2.3:45643)
I20260619 14:36:10.534664 21736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:10.529866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.3:45643 (local address 127.2.2.2:49351)
0619 14:36:10.530285 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:10.530296 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:36:10.530371 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:36:10.530627 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:36:10.530630 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:36:10.530647 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:36:10.530776 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:10.530781 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:10.531717 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:10.531721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:10.533259 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:10.533275 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:10.534233 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:10.534236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:10.534340 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:10.534363 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 14:36:10.534428 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 14:36:10.534493 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":133,"thread_start_us":136,"threads_started":1}
I20260619 14:36:10.534708 21725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:10.530035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:49351 (local address 127.2.2.3:45643)
0619 14:36:10.530178 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:10.530183 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:10.530388 (+ 205us) server_negotiation.cc:407] Connection header received
0619 14:36:10.530470 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:10.530473 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:10.530526 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:10.530599 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:36:10.530885 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:10.531609 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:10.533431 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:10.534063 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:10.534228 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:10.534295 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:10.534471 (+ 176us) server_negotiation.cc:299] Negotiation successful
0619 14:36:10.534573 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 14:36:10.534818 21578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:10.529798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.1:36067 (local address 127.2.2.2:41365)
0619 14:36:10.529918 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:10.529934 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:36:10.530025 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:36:10.530461 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:36:10.530466 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:36:10.530479 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:36:10.530615 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:10.530622 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:10.531717 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:10.531721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:10.533286 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:10.533293 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:10.534370 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:10.534373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:10.534517 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:10.534541 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 14:36:10.534622 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 14:36:10.534696 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":157}
I20260619 14:36:10.534878 21723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:10.530042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:41365 (local address 127.2.2.1:36067)
0619 14:36:10.530197 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:10.530204 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:10.530219 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:36:10.530276 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:10.530279 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:10.530319 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:10.530428 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:36:10.530736 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:10.531576 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:10.533431 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:10.534098 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:10.534228 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:10.534295 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:10.534662 (+ 367us) server_negotiation.cc:299] Negotiation successful
0619 14:36:10.534752 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 14:36:10.535583 21396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d98a21a4822e4fb2b054efe1eb6c0354" candidate_uuid: "97f600aa840d4439a6e1c6a2d541ebe4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ca2ce1807214f85b25ca04d9e8e2918" is_pre_election: true
I20260619 14:36:10.535754 21664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d98a21a4822e4fb2b054efe1eb6c0354" candidate_uuid: "97f600aa840d4439a6e1c6a2d541ebe4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "504012ca24084b3ba039993e0ae2fa89" is_pre_election: true
I20260619 14:36:10.535946 21396 raft_consensus.cc:2454] T d98a21a4822e4fb2b054efe1eb6c0354 P 3ca2ce1807214f85b25ca04d9e8e2918 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97f600aa840d4439a6e1c6a2d541ebe4 in term 0.
I20260619 14:36:10.535985 21664 raft_consensus.cc:2454] T d98a21a4822e4fb2b054efe1eb6c0354 P 504012ca24084b3ba039993e0ae2fa89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97f600aa840d4439a6e1c6a2d541ebe4 in term 0.
I20260619 14:36:10.536465 21463 leader_election.cc:302] T d98a21a4822e4fb2b054efe1eb6c0354 P 97f600aa840d4439a6e1c6a2d541ebe4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ca2ce1807214f85b25ca04d9e8e2918, 97f600aa840d4439a6e1c6a2d541ebe4; no voters:
I20260619 14:36:10.536768 21735 raft_consensus.cc:2781] T d98a21a4822e4fb2b054efe1eb6c0354 P 97f600aa840d4439a6e1c6a2d541ebe4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:36:10.536851 21735 raft_consensus.cc:482] T d98a21a4822e4fb2b054efe1eb6c0354 P 97f600aa840d4439a6e1c6a2d541ebe4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:36:10.536881 21735 raft_consensus.cc:3037] T d98a21a4822e4fb2b054efe1eb6c0354 P 97f600aa840d4439a6e1c6a2d541ebe4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:36:10.537976 21735 raft_consensus.cc:504] T d98a21a4822e4fb2b054efe1eb6c0354 P 97f600aa840d4439a6e1c6a2d541ebe4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ca2ce1807214f85b25ca04d9e8e2918" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 36067 } } peers { permanent_uuid: "97f600aa840d4439a6e1c6a2d541ebe4" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 43217 } } peers { permanent_uuid: "504012ca24084b3ba039993e0ae2fa89" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45643 } }
I20260619 14:36:10.538378 21735 leader_election.cc:288] T d98a21a4822e4fb2b054efe1eb6c0354 P 97f600aa840d4439a6e1c6a2d541ebe4 [CANDIDATE]: Term 1 election: Requested vote from peers 3ca2ce1807214f85b25ca04d9e8e2918 (127.2.2.1:36067), 504012ca24084b3ba039993e0ae2fa89 (127.2.2.3:45643)
I20260619 14:36:10.538803 21396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d98a21a4822e4fb2b054efe1eb6c0354" candidate_uuid: "97f600aa840d4439a6e1c6a2d541ebe4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ca2ce1807214f85b25ca04d9e8e2918"
I20260619 14:36:10.538811 21664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d98a21a4822e4fb2b054efe1eb6c0354" candidate_uuid: "97f600aa840d4439a6e1c6a2d541ebe4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "504012ca24084b3ba039993e0ae2fa89"
I20260619 14:36:10.538915 21396 raft_consensus.cc:3037] T d98a21a4822e4fb2b054efe1eb6c0354 P 3ca2ce1807214f85b25ca04d9e8e2918 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:36:10.538945 21664 raft_consensus.cc:3037] T d98a21a4822e4fb2b054efe1eb6c0354 P 504012ca24084b3ba039993e0ae2fa89 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:36:10.540095 21396 raft_consensus.cc:2454] T d98a21a4822e4fb2b054efe1eb6c0354 P 3ca2ce1807214f85b25ca04d9e8e2918 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97f600aa840d4439a6e1c6a2d541ebe4 in term 1.
I20260619 14:36:10.540198 21664 raft_consensus.cc:2454] T d98a21a4822e4fb2b054efe1eb6c0354 P 504012ca24084b3ba039993e0ae2fa89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97f600aa840d4439a6e1c6a2d541ebe4 in term 1.
I20260619 14:36:10.540443 21463 leader_election.cc:302] T d98a21a4822e4fb2b054efe1eb6c0354 P 97f600aa840d4439a6e1c6a2d541ebe4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ca2ce1807214f85b25ca04d9e8e2918, 97f600aa840d4439a6e1c6a2d541ebe4; no voters:
I20260619 14:36:10.540619 21735 raft_consensus.cc:2781] T d98a21a4822e4fb2b054efe1eb6c0354 P 97f600aa840d4439a6e1c6a2d541ebe4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:36:10.540868 21735 raft_consensus.cc:686] T d98a21a4822e4fb2b054efe1eb6c0354 P 97f600aa840d4439a6e1c6a2d541ebe4 [term 1 LEADER]: Becoming Leader. State: Replica: 97f600aa840d4439a6e1c6a2d541ebe4, State: Running, Role: LEADER
I20260619 14:36:10.541265 21735 consensus_queue.cc:237] T d98a21a4822e4fb2b054efe1eb6c0354 P 97f600aa840d4439a6e1c6a2d541ebe4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ca2ce1807214f85b25ca04d9e8e2918" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 36067 } } peers { permanent_uuid: "97f600aa840d4439a6e1c6a2d541ebe4" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 43217 } } peers { permanent_uuid: "504012ca24084b3ba039993e0ae2fa89" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45643 } }
I20260619 14:36:10.544723 21249 catalog_manager.cc:5495] T d98a21a4822e4fb2b054efe1eb6c0354 P 97f600aa840d4439a6e1c6a2d541ebe4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 97f600aa840d4439a6e1c6a2d541ebe4 (127.2.2.2). New cstate: current_term: 1 leader_uuid: "97f600aa840d4439a6e1c6a2d541ebe4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ca2ce1807214f85b25ca04d9e8e2918" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 36067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97f600aa840d4439a6e1c6a2d541ebe4" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 43217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "504012ca24084b3ba039993e0ae2fa89" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45643 } health_report { overall_health: UNKNOWN } } }
I20260619 14:36:10.572549 21726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:10.569314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52610 (local address 127.2.2.2:43217)
0619 14:36:10.569438 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:10.569441 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:10.569454 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:36:10.569551 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:10.569553 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:10.569604 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:10.569701 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:36:10.569925 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:10.570444 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:10.571265 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:10.571474 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:10.571545 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:10.571813 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:36:10.572226 (+ 413us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:36:10.572315 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:10.572373 (+ 58us) server_negotiation.cc:299] Negotiation successful
0619 14:36:10.572420 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 14:36:10.578531 21396 raft_consensus.cc:1261] T d98a21a4822e4fb2b054efe1eb6c0354 P 3ca2ce1807214f85b25ca04d9e8e2918 [term 1 FOLLOWER]: Refusing update from remote peer 97f600aa840d4439a6e1c6a2d541ebe4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:36:10.578552 21664 raft_consensus.cc:1261] T d98a21a4822e4fb2b054efe1eb6c0354 P 504012ca24084b3ba039993e0ae2fa89 [term 1 FOLLOWER]: Refusing update from remote peer 97f600aa840d4439a6e1c6a2d541ebe4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:36:10.579299 21737 consensus_queue.cc:1035] T d98a21a4822e4fb2b054efe1eb6c0354 P 97f600aa840d4439a6e1c6a2d541ebe4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ca2ce1807214f85b25ca04d9e8e2918" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 36067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:36:10.579480 21735 consensus_queue.cc:1035] T d98a21a4822e4fb2b054efe1eb6c0354 P 97f600aa840d4439a6e1c6a2d541ebe4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "504012ca24084b3ba039993e0ae2fa89" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 45643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:36:10.587453 21740 mvcc.cc:205] Tried to move back new op lower bound from 7298579540282396672 to 7298579540139560960. Current Snapshot: MvccSnapshot[applied={T|T < 7298579540282396672 or (T in {7298579540282396672})}]
I20260619 14:36:10.587657 21741 mvcc.cc:205] Tried to move back new op lower bound from 7298579540282396672 to 7298579540139560960. Current Snapshot: MvccSnapshot[applied={T|T < 7298579540282396672}]
I20260619 14:36:10.590801 21742 mvcc.cc:205] Tried to move back new op lower bound from 7298579540282396672 to 7298579540139560960. Current Snapshot: MvccSnapshot[applied={T|T < 7298579540282396672}]
I20260619 14:36:10.594993 21249 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:36:10.595171 21249 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:36:10.597517 21249 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4c2e598d45824b608635586ee8618e69: Sending DeleteTablet for 3 replicas of tablet d98a21a4822e4fb2b054efe1eb6c0354
I20260619 14:36:10.598297 21376 tablet_service.cc:1514] Processing DeleteTablet for tablet d98a21a4822e4fb2b054efe1eb6c0354 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:36:10 UTC) from {username='kudu', principal='kudu/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:60274
I20260619 14:36:10.598435 21510 tablet_service.cc:1514] Processing DeleteTablet for tablet d98a21a4822e4fb2b054efe1eb6c0354 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:36:10 UTC) from {username='kudu', principal='kudu/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:52604
I20260619 14:36:10.598584 21644 tablet_service.cc:1514] Processing DeleteTablet for tablet d98a21a4822e4fb2b054efe1eb6c0354 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:36:10 UTC) from {username='kudu', principal='kudu/127.2.2.62@KRBTEST.COM'} at 127.0.0.1:52094
I20260619 14:36:10.599080 21750 tablet_replica.cc:330] T d98a21a4822e4fb2b054efe1eb6c0354 P 3ca2ce1807214f85b25ca04d9e8e2918: stopping tablet replica
I20260619 14:36:10.599330 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 21313
I20260619 14:36:10.601137 21752 tablet_replica.cc:330] T d98a21a4822e4fb2b054efe1eb6c0354 P 504012ca24084b3ba039993e0ae2fa89: stopping tablet replica
I20260619 14:36:10.601795 21752 raft_consensus.cc:2229] T d98a21a4822e4fb2b054efe1eb6c0354 P 504012ca24084b3ba039993e0ae2fa89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:36:10.602175 21752 raft_consensus.cc:2258] T d98a21a4822e4fb2b054efe1eb6c0354 P 504012ca24084b3ba039993e0ae2fa89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:36:10.603840 21752 ts_tablet_manager.cc:1905] T d98a21a4822e4fb2b054efe1eb6c0354 P 504012ca24084b3ba039993e0ae2fa89: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 14:36:10.606835 21235 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.2.1:36067 (error 108)
I20260619 14:36:10.607012 21751 tablet_replica.cc:330] T d98a21a4822e4fb2b054efe1eb6c0354 P 97f600aa840d4439a6e1c6a2d541ebe4: stopping tablet replica
I20260619 14:36:10.607224 21752 ts_tablet_manager.cc:1918] T d98a21a4822e4fb2b054efe1eb6c0354 P 504012ca24084b3ba039993e0ae2fa89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:36:10.607331 21752 log.cc:1196] T d98a21a4822e4fb2b054efe1eb6c0354 P 504012ca24084b3ba039993e0ae2fa89: Deleting WAL directory at /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879514993456-2056-0/minicluster-data/ts-2/wal/wals/d98a21a4822e4fb2b054efe1eb6c0354
I20260619 14:36:10.607527 21751 raft_consensus.cc:2229] T d98a21a4822e4fb2b054efe1eb6c0354 P 97f600aa840d4439a6e1c6a2d541ebe4 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:36:10.607671 21752 ts_tablet_manager.cc:1939] T d98a21a4822e4fb2b054efe1eb6c0354 P 504012ca24084b3ba039993e0ae2fa89: Deleting consensus metadata
I20260619 14:36:10.607703 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 21447
I20260619 14:36:10.608029 21751 raft_consensus.cc:2258] T d98a21a4822e4fb2b054efe1eb6c0354 P 97f600aa840d4439a6e1c6a2d541ebe4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:36:10.609195 21724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:10.608800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.1:36067 (local address 127.0.0.1:60290)
0619 14:36:10.609004 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:10.609093 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.1:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260619 14:36:10.609278 21238 catalog_manager.cc:4841] TS 504012ca24084b3ba039993e0ae2fa89 (127.2.2.3:45643): tablet d98a21a4822e4fb2b054efe1eb6c0354 (table test-table [id=11f70167e2004a2f8e701e7d7d5f204e]) successfully deleted
W20260619 14:36:10.609381 21235 catalog_manager.cc:4584] TS 3ca2ce1807214f85b25ca04d9e8e2918 (127.2.2.1:36067): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d98a21a4822e4fb2b054efe1eb6c0354: Network error: Client connection negotiation failed: client connection to 127.2.2.1:36067: connect: Connection refused (error 111)
I20260619 14:36:10.614254 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 21581
I20260619 14:36:10.616001 21724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:10.615228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.2:43217 (local address 127.0.0.1:52624)
0619 14:36:10.615828 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:10.615896 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.2:43217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":498}
I20260619 14:36:10.620191 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 21219
2026-06-19T14:36:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2319 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-19T14:36:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:36:10Z Disabled control of system clock
I20260619 14:36:10.650691 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:36193
--webserver_interface=127.2.2.62
--webserver_port=0
--builtin_ntp_servers=127.2.2.20:43383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.2.62:36193
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 14:36:10.763857 21757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:36:10.764149 21757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:36:10.764185 21757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:36:10.768601 21757 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:36:10.768697 21757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:36:10.768713 21757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:36:10.768731 21757 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:36:10.768747 21757 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:36:10.773842 21757 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:43383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.2.62:36193
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:36193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.2.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:36:10.775028 21757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:36:10.776423 21757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:36:10.785521 21765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:10.785593 21757 server_base.cc:1034] running on GCE node
W20260619 14:36:10.785658 21762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:10.785521 21763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:10.786456 21757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:36:10.787706 21757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:36:10.788913 21757 hybrid_clock.cc:648] HybridClock initialized: now 1781879770788895 us; error 33 us; skew 500 ppm
I20260619 14:36:10.791109 21757 webserver.cc:460] Webserver started at http://127.2.2.62:42961/ using document root <none> and password file <none>
I20260619 14:36:10.791745 21757 fs_manager.cc:359] Metadata directory not provided
I20260619 14:36:10.791846 21757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:36:10.792093 21757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:36:10.794339 21757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/master-0/data/instance:
uuid: "904d0e5326114ca4877b0bf5790f681f"
format_stamp: "Formatted at 2026-06-19 14:36:10 on dist-test-slave-70rm"
server_key: "352765982f7603f7e77fe0914a8b733b"
server_key_iv: "441e2dba7db7fdea8865456bc62fd69f"
server_key_version: "encryptionkey@0"
I20260619 14:36:10.794950 21757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/master-0/wal/instance:
uuid: "904d0e5326114ca4877b0bf5790f681f"
format_stamp: "Formatted at 2026-06-19 14:36:10 on dist-test-slave-70rm"
server_key: "352765982f7603f7e77fe0914a8b733b"
server_key_iv: "441e2dba7db7fdea8865456bc62fd69f"
server_key_version: "encryptionkey@0"
I20260619 14:36:10.798923 21757 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 14:36:10.801514 21771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:10.802806 21757 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:36:10.802954 21757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/master-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/master-0/wal
uuid: "904d0e5326114ca4877b0bf5790f681f"
format_stamp: "Formatted at 2026-06-19 14:36:10 on dist-test-slave-70rm"
server_key: "352765982f7603f7e77fe0914a8b733b"
server_key_iv: "441e2dba7db7fdea8865456bc62fd69f"
server_key_version: "encryptionkey@0"
I20260619 14:36:10.803076 21757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:36:10.830473 21757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:36:10.831267 21757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:36:10.831476 21757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:36:10.838995 21757 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.62:36193
I20260619 14:36:10.839020 21823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.62:36193 every 8 connection(s)
I20260619 14:36:10.840066 21757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
I20260619 14:36:10.843799 21824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:36:10.846784 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 21757
I20260619 14:36:10.846937 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/master-0/wal/instance
I20260619 14:36:10.847288 2056 external_mini_cluster.cc:1428] Setting key 1f0d4fb2055c29ddcd55cabb60a15911
I20260619 14:36:10.850783 21824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 904d0e5326114ca4877b0bf5790f681f: Bootstrap starting.
I20260619 14:36:10.853317 21824 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 904d0e5326114ca4877b0bf5790f681f: Neither blocks nor log segments found. Creating new log.
I20260619 14:36:10.854207 21824 log.cc:824] T 00000000000000000000000000000000 P 904d0e5326114ca4877b0bf5790f681f: Log is configured to *not* fsync() on all Append() calls
I20260619 14:36:10.856750 21824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 904d0e5326114ca4877b0bf5790f681f: No bootstrap required, opened a new log
I20260619 14:36:10.857462 21827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:10.848963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35538 (local address 127.2.2.62:36193)
0619 14:36:10.849660 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:10.849675 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:10.849715 (+ 40us) server_negotiation.cc:407] Connection header received
0619 14:36:10.850421 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:10.850440 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:10.850890 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:10.851298 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:10.851784 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:10.852807 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:10.853781 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:10.854354 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:10.854523 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:10.855572 (+ 1049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:10.855577 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:10.855592 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:36:10.855998 (+ 406us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:10.856055 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:36:10.856075 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:10.856153 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:10.856342 (+ 189us) server_negotiation.cc:299] Negotiation successful
0619 14:36:10.856646 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":138,"threads_started":1}
I20260619 14:36:10.861023 21824 raft_consensus.cc:348] T 00000000000000000000000000000000 P 904d0e5326114ca4877b0bf5790f681f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "904d0e5326114ca4877b0bf5790f681f" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 36193 } }
I20260619 14:36:10.861317 21824 raft_consensus.cc:374] T 00000000000000000000000000000000 P 904d0e5326114ca4877b0bf5790f681f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:36:10.861400 21824 raft_consensus.cc:729] T 00000000000000000000000000000000 P 904d0e5326114ca4877b0bf5790f681f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 904d0e5326114ca4877b0bf5790f681f, State: Initialized, Role: FOLLOWER
I20260619 14:36:10.861797 21824 consensus_queue.cc:260] T 00000000000000000000000000000000 P 904d0e5326114ca4877b0bf5790f681f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "904d0e5326114ca4877b0bf5790f681f" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 36193 } }
I20260619 14:36:10.861923 21824 raft_consensus.cc:388] T 00000000000000000000000000000000 P 904d0e5326114ca4877b0bf5790f681f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:36:10.861976 21824 raft_consensus.cc:482] T 00000000000000000000000000000000 P 904d0e5326114ca4877b0bf5790f681f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:36:10.862056 21824 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 904d0e5326114ca4877b0bf5790f681f [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:36:10.863451 21824 raft_consensus.cc:504] T 00000000000000000000000000000000 P 904d0e5326114ca4877b0bf5790f681f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "904d0e5326114ca4877b0bf5790f681f" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 36193 } }
I20260619 14:36:10.863790 21824 leader_election.cc:302] T 00000000000000000000000000000000 P 904d0e5326114ca4877b0bf5790f681f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 904d0e5326114ca4877b0bf5790f681f; no voters:
I20260619 14:36:10.864084 21824 leader_election.cc:288] T 00000000000000000000000000000000 P 904d0e5326114ca4877b0bf5790f681f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:36:10.864221 21829 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 904d0e5326114ca4877b0bf5790f681f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:36:10.864488 21829 raft_consensus.cc:686] T 00000000000000000000000000000000 P 904d0e5326114ca4877b0bf5790f681f [term 1 LEADER]: Becoming Leader. State: Replica: 904d0e5326114ca4877b0bf5790f681f, State: Running, Role: LEADER
I20260619 14:36:10.864842 21829 consensus_queue.cc:237] T 00000000000000000000000000000000 P 904d0e5326114ca4877b0bf5790f681f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "904d0e5326114ca4877b0bf5790f681f" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 36193 } }
I20260619 14:36:10.865593 21824 sys_catalog.cc:564] T 00000000000000000000000000000000 P 904d0e5326114ca4877b0bf5790f681f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:36:10.866532 21831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 904d0e5326114ca4877b0bf5790f681f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 904d0e5326114ca4877b0bf5790f681f. Latest consensus state: current_term: 1 leader_uuid: "904d0e5326114ca4877b0bf5790f681f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "904d0e5326114ca4877b0bf5790f681f" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 36193 } } }
I20260619 14:36:10.866677 21831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 904d0e5326114ca4877b0bf5790f681f [sys.catalog]: This master's current role is: LEADER
I20260619 14:36:10.866981 21830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 904d0e5326114ca4877b0bf5790f681f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "904d0e5326114ca4877b0bf5790f681f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "904d0e5326114ca4877b0bf5790f681f" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 36193 } } }
I20260619 14:36:10.867089 21830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 904d0e5326114ca4877b0bf5790f681f [sys.catalog]: This master's current role is: LEADER
I20260619 14:36:10.867178 21836 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:36:10.870579 21836 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:36:10.876664 21836 catalog_manager.cc:1347] Generated new cluster ID: e6da5625cd9c428d8f2b6f77736a7c2d
I20260619 14:36:10.876768 21836 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:36:10.903713 21836 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:36:10.905072 21836 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:36:10.912688 21836 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 904d0e5326114ca4877b0bf5790f681f: Generated new TSK 0
I20260619 14:36:10.913661 21836 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:36:10.919464 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.1:0
--local_ip_for_outbound_sockets=127.2.2.1
--webserver_interface=127.2.2.1
--webserver_port=0
--tserver_master_addrs=127.2.2.62:36193
--builtin_ntp_servers=127.2.2.20:43383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 14:36:11.035068 21848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:36:11.035385 21848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:36:11.035461 21848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:36:11.039624 21848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:36:11.039721 21848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:36:11.039815 21848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.1
I20260619 14:36:11.044848 21848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:43383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.2.1
--webserver_port=0
--tserver_master_addrs=127.2.2.62:36193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.1
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:36:11.046048 21848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:36:11.047529 21848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:36:11.057945 21853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:11.057992 21854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:11.057960 21856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:11.057971 21848 server_base.cc:1034] running on GCE node
I20260619 14:36:11.058756 21848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:36:11.059455 21848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:36:11.060660 21848 hybrid_clock.cc:648] HybridClock initialized: now 1781879771060624 us; error 61 us; skew 500 ppm
I20260619 14:36:11.062706 21848 webserver.cc:460] Webserver started at http://127.2.2.1:45527/ using document root <none> and password file <none>
I20260619 14:36:11.063349 21848 fs_manager.cc:359] Metadata directory not provided
I20260619 14:36:11.063488 21848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:36:11.063740 21848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:36:11.066057 21848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/ts-0/data/instance:
uuid: "1c6bbb47e8c24b01a83a6009bc4c8af9"
format_stamp: "Formatted at 2026-06-19 14:36:11 on dist-test-slave-70rm"
server_key: "ba6e2c18ab1df20dfe9e95e1c7b8afe0"
server_key_iv: "605e944ab33b6ca5236424745f37decf"
server_key_version: "encryptionkey@0"
I20260619 14:36:11.066717 21848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/ts-0/wal/instance:
uuid: "1c6bbb47e8c24b01a83a6009bc4c8af9"
format_stamp: "Formatted at 2026-06-19 14:36:11 on dist-test-slave-70rm"
server_key: "ba6e2c18ab1df20dfe9e95e1c7b8afe0"
server_key_iv: "605e944ab33b6ca5236424745f37decf"
server_key_version: "encryptionkey@0"
I20260619 14:36:11.070981 21848 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:36:11.073720 21862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:11.075172 21848 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:36:11.075331 21848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/ts-0/wal
uuid: "1c6bbb47e8c24b01a83a6009bc4c8af9"
format_stamp: "Formatted at 2026-06-19 14:36:11 on dist-test-slave-70rm"
server_key: "ba6e2c18ab1df20dfe9e95e1c7b8afe0"
server_key_iv: "605e944ab33b6ca5236424745f37decf"
server_key_version: "encryptionkey@0"
I20260619 14:36:11.075474 21848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:36:11.094329 21848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:36:11.095103 21848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:36:11.095323 21848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:36:11.095952 21848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:36:11.096944 21848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:36:11.096992 21848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:11.097033 21848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:36:11.097048 21848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:11.106281 21848 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.1:46817
I20260619 14:36:11.106313 21975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.1:46817 every 8 connection(s)
I20260619 14:36:11.107486 21848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/ts-0/data/info.pb
I20260619 14:36:11.116613 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 21848
I20260619 14:36:11.116767 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/ts-0/wal/instance
I20260619 14:36:11.117105 2056 external_mini_cluster.cc:1428] Setting key 904406328137d827d4b4bfcbed9285ca
I20260619 14:36:11.117919 21827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:11.109609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:41923 (local address 127.2.2.62:36193)
0619 14:36:11.109763 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:11.109767 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:11.110981 (+ 1214us) server_negotiation.cc:407] Connection header received
0619 14:36:11.112499 (+ 1518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:11.112502 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:11.112574 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:11.112681 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:11.113642 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:11.114275 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:11.115481 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:11.115725 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:11.115846 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:11.117315 (+ 1469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:11.117319 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:11.117322 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:36:11.117344 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:11.117367 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:36:11.117377 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:11.117457 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:11.117682 (+ 225us) server_negotiation.cc:299] Negotiation successful
0619 14:36:11.117771 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 14:36:11.119028 21978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:11.110014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:36193 (local address 127.2.2.1:41923)
0619 14:36:11.110815 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:11.110858 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 14:36:11.112235 (+ 1377us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:36:11.112876 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:36:11.112883 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:36:11.112974 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:36:11.113485 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:11.113498 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:11.114413 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:11.114417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:11.115341 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:11.115351 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:11.115832 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:11.115909 (+ 77us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:11.116093 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:11.116874 (+ 781us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:36:11.116877 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:36:11.116948 (+ 71us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:36:11.117048 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:36:11.117482 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:36:11.117493 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:36:11.117499 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 14:36:11.117678 (+ 179us) client_negotiation.cc:240] Negotiation successful
0619 14:36:11.118012 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":146,"threads_started":1}
I20260619 14:36:11.119433 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.2:0
--local_ip_for_outbound_sockets=127.2.2.2
--webserver_interface=127.2.2.2
--webserver_port=0
--tserver_master_addrs=127.2.2.62:36193
--builtin_ntp_servers=127.2.2.20:43383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 14:36:11.120649 21976 heartbeater.cc:347] Connected to a master server at 127.2.2.62:36193
I20260619 14:36:11.120983 21976 heartbeater.cc:464] Registering TS with master...
I20260619 14:36:11.121716 21976 heartbeater.cc:511] Master 127.2.2.62:36193 requested a full tablet report, sending...
I20260619 14:36:11.123577 21788 ts_manager.cc:194] Registered new tserver with Master: 1c6bbb47e8c24b01a83a6009bc4c8af9 (127.2.2.1:46817)
I20260619 14:36:11.124809 21788 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.2.1:41923
W20260619 14:36:11.235620 21979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:36:11.235898 21979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:36:11.235936 21979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:36:11.240715 21979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:36:11.240782 21979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:36:11.240867 21979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.2
I20260619 14:36:11.245750 21979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:43383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.2.2
--webserver_port=0
--tserver_master_addrs=127.2.2.62:36193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.2
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:36:11.246912 21979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:36:11.248310 21979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:36:11.258239 21984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:11.258242 21987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:11.258242 21985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:11.258646 21979 server_base.cc:1034] running on GCE node
I20260619 14:36:11.259093 21979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:36:11.259724 21979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:36:11.260890 21979 hybrid_clock.cc:648] HybridClock initialized: now 1781879771260872 us; error 30 us; skew 500 ppm
I20260619 14:36:11.263007 21979 webserver.cc:460] Webserver started at http://127.2.2.2:38981/ using document root <none> and password file <none>
I20260619 14:36:11.263641 21979 fs_manager.cc:359] Metadata directory not provided
I20260619 14:36:11.263708 21979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:36:11.263967 21979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:36:11.266290 21979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/ts-1/data/instance:
uuid: "c15ac629bf95489f81863e9c1dc817cb"
format_stamp: "Formatted at 2026-06-19 14:36:11 on dist-test-slave-70rm"
server_key: "0c991495ae0e94bcd1ca37b4986125e2"
server_key_iv: "f49bb3312f41d92229b2992df091d1d7"
server_key_version: "encryptionkey@0"
I20260619 14:36:11.267006 21979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/ts-1/wal/instance:
uuid: "c15ac629bf95489f81863e9c1dc817cb"
format_stamp: "Formatted at 2026-06-19 14:36:11 on dist-test-slave-70rm"
server_key: "0c991495ae0e94bcd1ca37b4986125e2"
server_key_iv: "f49bb3312f41d92229b2992df091d1d7"
server_key_version: "encryptionkey@0"
I20260619 14:36:11.271076 21979 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 14:36:11.273770 21993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:11.275115 21979 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 14:36:11.275267 21979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/ts-1/wal
uuid: "c15ac629bf95489f81863e9c1dc817cb"
format_stamp: "Formatted at 2026-06-19 14:36:11 on dist-test-slave-70rm"
server_key: "0c991495ae0e94bcd1ca37b4986125e2"
server_key_iv: "f49bb3312f41d92229b2992df091d1d7"
server_key_version: "encryptionkey@0"
I20260619 14:36:11.275385 21979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:36:11.304582 21979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:36:11.305377 21979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:36:11.305589 21979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:36:11.306315 21979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:36:11.307380 21979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:36:11.307437 21979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:11.307502 21979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:36:11.307538 21979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:11.316629 21979 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.2:46409
I20260619 14:36:11.316645 22106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.2:46409 every 8 connection(s)
I20260619 14:36:11.317646 21979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/ts-1/data/info.pb
I20260619 14:36:11.327131 21827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:11.319564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.2:38233 (local address 127.2.2.62:36193)
0619 14:36:11.319711 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:11.319715 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:11.320799 (+ 1084us) server_negotiation.cc:407] Connection header received
0619 14:36:11.322273 (+ 1474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:11.322276 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:11.322337 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:11.322444 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:11.323377 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:11.323917 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:11.325094 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:11.325325 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:11.325439 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:11.326615 (+ 1176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:11.326619 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:11.326621 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:36:11.326643 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:11.326662 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:36:11.326671 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:11.326754 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:11.326913 (+ 159us) server_negotiation.cc:299] Negotiation successful
0619 14:36:11.326980 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 14:36:11.327142 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 21979
I20260619 14:36:11.327288 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/ts-1/wal/instance
I20260619 14:36:11.327625 2056 external_mini_cluster.cc:1428] Setting key 26b33ebf8424be96fbe01d9eb24b0fc8
I20260619 14:36:11.328091 22109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:11.319989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:36193 (local address 127.2.2.2:38233)
0619 14:36:11.320641 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:11.320682 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 14:36:11.322006 (+ 1324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:36:11.322633 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:36:11.322640 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:36:11.322736 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:36:11.323231 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:11.323243 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:11.324049 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:11.324053 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:11.324961 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:11.324970 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:11.325427 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:11.325489 (+ 62us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:11.325655 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:11.326346 (+ 691us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:36:11.326349 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:36:11.326389 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:36:11.326454 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:36:11.326755 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:36:11.326766 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:36:11.326775 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 14:36:11.326914 (+ 139us) client_negotiation.cc:240] Negotiation successful
0619 14:36:11.327208 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":105,"threads_started":1}
I20260619 14:36:11.329373 22107 heartbeater.cc:347] Connected to a master server at 127.2.2.62:36193
I20260619 14:36:11.329627 22107 heartbeater.cc:464] Registering TS with master...
I20260619 14:36:11.329818 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.2.3:0
--local_ip_for_outbound_sockets=127.2.2.3
--webserver_interface=127.2.2.3
--webserver_port=0
--tserver_master_addrs=127.2.2.62:36193
--builtin_ntp_servers=127.2.2.20:43383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 14:36:11.330298 22107 heartbeater.cc:511] Master 127.2.2.62:36193 requested a full tablet report, sending...
I20260619 14:36:11.331463 21788 ts_manager.cc:194] Registered new tserver with Master: c15ac629bf95489f81863e9c1dc817cb (127.2.2.2:46409)
I20260619 14:36:11.332034 21788 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.2.2:38233
W20260619 14:36:11.448074 22110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:36:11.448369 22110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:36:11.448410 22110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:36:11.452585 22110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:36:11.452656 22110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:36:11.452762 22110 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.2.3
I20260619 14:36:11.457772 22110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:43383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.2.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.2.3
--webserver_port=0
--tserver_master_addrs=127.2.2.62:36193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.2.3
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:36:11.458945 22110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:36:11.460374 22110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:36:11.470445 22116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:11.470494 22115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:11.470527 22118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:11.470666 22110 server_base.cc:1034] running on GCE node
I20260619 14:36:11.471138 22110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:36:11.471889 22110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:36:11.473119 22110 hybrid_clock.cc:648] HybridClock initialized: now 1781879771473085 us; error 55 us; skew 500 ppm
I20260619 14:36:11.475193 22110 webserver.cc:460] Webserver started at http://127.2.2.3:33705/ using document root <none> and password file <none>
I20260619 14:36:11.475806 22110 fs_manager.cc:359] Metadata directory not provided
I20260619 14:36:11.475905 22110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:36:11.476147 22110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:36:11.478299 22110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/ts-2/data/instance:
uuid: "858c49fb1c43456eb350236121c9b2e2"
format_stamp: "Formatted at 2026-06-19 14:36:11 on dist-test-slave-70rm"
server_key: "ce8f634ecb11904e9169932f07ee4605"
server_key_iv: "82a3679e7f3fd18637bddc2f1ef25f34"
server_key_version: "encryptionkey@0"
I20260619 14:36:11.478978 22110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/ts-2/wal/instance:
uuid: "858c49fb1c43456eb350236121c9b2e2"
format_stamp: "Formatted at 2026-06-19 14:36:11 on dist-test-slave-70rm"
server_key: "ce8f634ecb11904e9169932f07ee4605"
server_key_iv: "82a3679e7f3fd18637bddc2f1ef25f34"
server_key_version: "encryptionkey@0"
I20260619 14:36:11.483083 22110 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 14:36:11.485559 22124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:11.486964 22110 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 14:36:11.487104 22110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/ts-2/wal
uuid: "858c49fb1c43456eb350236121c9b2e2"
format_stamp: "Formatted at 2026-06-19 14:36:11 on dist-test-slave-70rm"
server_key: "ce8f634ecb11904e9169932f07ee4605"
server_key_iv: "82a3679e7f3fd18637bddc2f1ef25f34"
server_key_version: "encryptionkey@0"
I20260619 14:36:11.487233 22110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:36:11.505290 22110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:36:11.506071 22110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:36:11.506330 22110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:36:11.507019 22110 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:36:11.508064 22110 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:36:11.508111 22110 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:11.508176 22110 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:36:11.508214 22110 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:11.517133 22110 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.3:36469
I20260619 14:36:11.517153 22237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.3:36469 every 8 connection(s)
I20260619 14:36:11.518064 22110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/ts-2/data/info.pb
I20260619 14:36:11.526980 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 22110
I20260619 14:36:11.527118 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/ts-2/wal/instance
I20260619 14:36:11.527438 21827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:11.519925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.3:44137 (local address 127.2.2.62:36193)
0619 14:36:11.520067 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:11.520071 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:11.521132 (+ 1061us) server_negotiation.cc:407] Connection header received
0619 14:36:11.522628 (+ 1496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:11.522632 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:11.522690 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:11.522798 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:11.523698 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:11.524251 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:11.525402 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:11.525637 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:11.525750 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:11.526946 (+ 1196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:11.526949 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:11.526952 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:36:11.526965 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:11.526986 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:36:11.526994 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:11.527087 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:11.527262 (+ 175us) server_negotiation.cc:299] Negotiation successful
0619 14:36:11.527321 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 14:36:11.527444 2056 external_mini_cluster.cc:1428] Setting key e4a54964e13bba64bb43b9052dc46c2f
I20260619 14:36:11.528637 22240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:11.520335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:36193 (local address 127.2.2.3:44137)
0619 14:36:11.520979 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:11.521019 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 14:36:11.522398 (+ 1379us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:36:11.522973 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:36:11.522980 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:36:11.523068 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:36:11.523553 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:11.523565 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:11.524390 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:11.524393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:11.525273 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:11.525282 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:11.525745 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:11.525805 (+ 60us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:11.525974 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:11.526620 (+ 646us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:36:11.526621 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:36:11.526661 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:36:11.526769 (+ 108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:36:11.527086 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:36:11.527096 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:36:11.527104 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 14:36:11.527242 (+ 138us) client_negotiation.cc:240] Negotiation successful
0619 14:36:11.527590 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":108,"threads_started":1}
I20260619 14:36:11.529779 22238 heartbeater.cc:347] Connected to a master server at 127.2.2.62:36193
I20260619 14:36:11.529992 22238 heartbeater.cc:464] Registering TS with master...
I20260619 14:36:11.530534 22238 heartbeater.cc:511] Master 127.2.2.62:36193 requested a full tablet report, sending...
I20260619 14:36:11.531466 21788 ts_manager.cc:194] Registered new tserver with Master: 858c49fb1c43456eb350236121c9b2e2 (127.2.2.3:36469)
I20260619 14:36:11.532048 21788 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.2.3:44137
I20260619 14:36:11.541764 2056 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:36:11.547308 21827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:11.543742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35544 (local address 127.2.2.62:36193)
0619 14:36:11.543877 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:11.543882 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:11.544005 (+ 123us) server_negotiation.cc:407] Connection header received
0619 14:36:11.544109 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:11.544111 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:11.544153 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:11.544218 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:11.544464 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:11.544980 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:11.545762 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:11.545962 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:11.546035 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:11.546879 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:11.546882 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:11.546884 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:36:11.546895 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:11.546912 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:36:11.546919 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:11.546974 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:11.547066 (+ 92us) server_negotiation.cc:299] Negotiation successful
0619 14:36:11.547115 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 14:36:11.550336 21788 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35544:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:36:11.553380 21788 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:36:11.568040 22249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:11.562422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.3:36469 (local address 127.0.0.1:51746)
0619 14:36:11.562835 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:11.562893 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 14:36:11.563036 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:36:11.563609 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:36:11.563617 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:36:11.563646 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:36:11.563836 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:11.563843 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:11.565314 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:11.565320 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:11.566303 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:11.566312 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:11.567454 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:11.567458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:11.567593 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:11.567639 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 14:36:11.567755 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 14:36:11.567883 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"spinlock_wait_cycles":11136,"thread_start_us":119,"threads_started":1}
I20260619 14:36:11.568071 22250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:11.562586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51746 (local address 127.2.2.3:36469)
0619 14:36:11.562879 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:11.562892 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:11.562968 (+ 76us) server_negotiation.cc:407] Connection header received
0619 14:36:11.563200 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:11.563207 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:11.563431 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:11.563992 (+ 561us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:36:11.564136 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:11.565202 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:11.566435 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:11.567323 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:11.567424 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:11.567636 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:11.567803 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 14:36:11.567906 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":111,"threads_started":1}
I20260619 14:36:11.569281 22252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:11.563200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.2:46409 (local address 127.0.0.1:55614)
0619 14:36:11.563699 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:11.563713 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:36:11.563818 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:36:11.564565 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:36:11.564570 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:36:11.564587 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:36:11.564712 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:11.564729 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:11.566404 (+ 1675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:11.566408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:11.567481 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:11.567490 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:11.568926 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:11.568929 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:11.569003 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:11.569021 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 14:36:11.569076 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 14:36:11.569127 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":79,"threads_started":1}
I20260619 14:36:11.569401 22253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:11.563482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55614 (local address 127.2.2.2:46409)
0619 14:36:11.564137 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:11.564148 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:11.564171 (+ 23us) server_negotiation.cc:407] Connection header received
0619 14:36:11.564241 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:11.564245 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:11.564418 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:11.564567 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:36:11.564851 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:11.566083 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:11.567611 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:11.568813 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:11.568917 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:11.569088 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:11.569139 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 14:36:11.569243 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":95,"threads_started":1}
I20260619 14:36:11.569932 22251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:11.563013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.1:46817 (local address 127.0.0.1:46538)
0619 14:36:11.563581 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:11.563605 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 14:36:11.563676 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:36:11.564717 (+ 1041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:36:11.564720 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:36:11.564746 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:36:11.564829 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:11.564835 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:11.566255 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:11.566259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:11.567434 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:11.567443 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:11.569282 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:11.569285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:11.569375 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:11.569400 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 14:36:11.569737 (+ 337us) client_negotiation.cc:240] Negotiation successful
0619 14:36:11.569809 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"mutex_wait_us":4,"thread_start_us":110,"threads_started":1}
I20260619 14:36:11.570401 22254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:11.563836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46538 (local address 127.2.2.1:46817)
0619 14:36:11.564299 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:11.564308 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:11.564327 (+ 19us) server_negotiation.cc:407] Connection header received
0619 14:36:11.564387 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:11.564390 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:11.564560 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:11.564733 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:36:11.564941 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:11.566076 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:11.567561 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:11.568406 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:11.568516 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:11.568726 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:11.570077 (+ 1351us) server_negotiation.cc:299] Negotiation successful
0619 14:36:11.570239 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":117,"threads_started":1}
I20260619 14:36:11.571564 22172 tablet_service.cc:1467] Processing CreateTablet for tablet 28bdf6ab838141fd8a383ec3011681a9 (DEFAULT_TABLE table=test-table [id=3c0e8f1865614db49c60710f75430357]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:36:11.572122 22041 tablet_service.cc:1467] Processing CreateTablet for tablet 28bdf6ab838141fd8a383ec3011681a9 (DEFAULT_TABLE table=test-table [id=3c0e8f1865614db49c60710f75430357]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:36:11.572696 22172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28bdf6ab838141fd8a383ec3011681a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:36:11.573043 22041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28bdf6ab838141fd8a383ec3011681a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:36:11.572855 21910 tablet_service.cc:1467] Processing CreateTablet for tablet 28bdf6ab838141fd8a383ec3011681a9 (DEFAULT_TABLE table=test-table [id=3c0e8f1865614db49c60710f75430357]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:36:11.573724 21910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28bdf6ab838141fd8a383ec3011681a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:36:11.579690 22255 tablet_bootstrap.cc:492] T 28bdf6ab838141fd8a383ec3011681a9 P c15ac629bf95489f81863e9c1dc817cb: Bootstrap starting.
I20260619 14:36:11.581177 22257 tablet_bootstrap.cc:492] T 28bdf6ab838141fd8a383ec3011681a9 P 1c6bbb47e8c24b01a83a6009bc4c8af9: Bootstrap starting.
I20260619 14:36:11.582554 22256 tablet_bootstrap.cc:492] T 28bdf6ab838141fd8a383ec3011681a9 P 858c49fb1c43456eb350236121c9b2e2: Bootstrap starting.
I20260619 14:36:11.583601 22257 tablet_bootstrap.cc:654] T 28bdf6ab838141fd8a383ec3011681a9 P 1c6bbb47e8c24b01a83a6009bc4c8af9: Neither blocks nor log segments found. Creating new log.
I20260619 14:36:11.583604 22255 tablet_bootstrap.cc:654] T 28bdf6ab838141fd8a383ec3011681a9 P c15ac629bf95489f81863e9c1dc817cb: Neither blocks nor log segments found. Creating new log.
I20260619 14:36:11.584523 22257 log.cc:824] T 28bdf6ab838141fd8a383ec3011681a9 P 1c6bbb47e8c24b01a83a6009bc4c8af9: Log is configured to *not* fsync() on all Append() calls
I20260619 14:36:11.585534 22255 log.cc:824] T 28bdf6ab838141fd8a383ec3011681a9 P c15ac629bf95489f81863e9c1dc817cb: Log is configured to *not* fsync() on all Append() calls
I20260619 14:36:11.585597 22256 tablet_bootstrap.cc:654] T 28bdf6ab838141fd8a383ec3011681a9 P 858c49fb1c43456eb350236121c9b2e2: Neither blocks nor log segments found. Creating new log.
I20260619 14:36:11.586954 22256 log.cc:824] T 28bdf6ab838141fd8a383ec3011681a9 P 858c49fb1c43456eb350236121c9b2e2: Log is configured to *not* fsync() on all Append() calls
I20260619 14:36:11.587339 22257 tablet_bootstrap.cc:492] T 28bdf6ab838141fd8a383ec3011681a9 P 1c6bbb47e8c24b01a83a6009bc4c8af9: No bootstrap required, opened a new log
I20260619 14:36:11.587546 22255 tablet_bootstrap.cc:492] T 28bdf6ab838141fd8a383ec3011681a9 P c15ac629bf95489f81863e9c1dc817cb: No bootstrap required, opened a new log
I20260619 14:36:11.587538 22257 ts_tablet_manager.cc:1397] T 28bdf6ab838141fd8a383ec3011681a9 P 1c6bbb47e8c24b01a83a6009bc4c8af9: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260619 14:36:11.587707 22255 ts_tablet_manager.cc:1397] T 28bdf6ab838141fd8a383ec3011681a9 P c15ac629bf95489f81863e9c1dc817cb: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260619 14:36:11.590008 22256 tablet_bootstrap.cc:492] T 28bdf6ab838141fd8a383ec3011681a9 P 858c49fb1c43456eb350236121c9b2e2: No bootstrap required, opened a new log
I20260619 14:36:11.590231 22256 ts_tablet_manager.cc:1397] T 28bdf6ab838141fd8a383ec3011681a9 P 858c49fb1c43456eb350236121c9b2e2: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260619 14:36:11.592564 22257 raft_consensus.cc:348] T 28bdf6ab838141fd8a383ec3011681a9 P 1c6bbb47e8c24b01a83a6009bc4c8af9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "858c49fb1c43456eb350236121c9b2e2" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 36469 } } peers { permanent_uuid: "c15ac629bf95489f81863e9c1dc817cb" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 46409 } } peers { permanent_uuid: "1c6bbb47e8c24b01a83a6009bc4c8af9" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 46817 } }
I20260619 14:36:11.592574 22255 raft_consensus.cc:348] T 28bdf6ab838141fd8a383ec3011681a9 P c15ac629bf95489f81863e9c1dc817cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "858c49fb1c43456eb350236121c9b2e2" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 36469 } } peers { permanent_uuid: "c15ac629bf95489f81863e9c1dc817cb" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 46409 } } peers { permanent_uuid: "1c6bbb47e8c24b01a83a6009bc4c8af9" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 46817 } }
I20260619 14:36:11.592846 22255 raft_consensus.cc:374] T 28bdf6ab838141fd8a383ec3011681a9 P c15ac629bf95489f81863e9c1dc817cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:36:11.592846 22257 raft_consensus.cc:374] T 28bdf6ab838141fd8a383ec3011681a9 P 1c6bbb47e8c24b01a83a6009bc4c8af9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:36:11.592898 22255 raft_consensus.cc:729] T 28bdf6ab838141fd8a383ec3011681a9 P c15ac629bf95489f81863e9c1dc817cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c15ac629bf95489f81863e9c1dc817cb, State: Initialized, Role: FOLLOWER
I20260619 14:36:11.592898 22257 raft_consensus.cc:729] T 28bdf6ab838141fd8a383ec3011681a9 P 1c6bbb47e8c24b01a83a6009bc4c8af9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c6bbb47e8c24b01a83a6009bc4c8af9, State: Initialized, Role: FOLLOWER
I20260619 14:36:11.593370 22257 consensus_queue.cc:260] T 28bdf6ab838141fd8a383ec3011681a9 P 1c6bbb47e8c24b01a83a6009bc4c8af9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "858c49fb1c43456eb350236121c9b2e2" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 36469 } } peers { permanent_uuid: "c15ac629bf95489f81863e9c1dc817cb" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 46409 } } peers { permanent_uuid: "1c6bbb47e8c24b01a83a6009bc4c8af9" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 46817 } }
I20260619 14:36:11.593370 22255 consensus_queue.cc:260] T 28bdf6ab838141fd8a383ec3011681a9 P c15ac629bf95489f81863e9c1dc817cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "858c49fb1c43456eb350236121c9b2e2" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 36469 } } peers { permanent_uuid: "c15ac629bf95489f81863e9c1dc817cb" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 46409 } } peers { permanent_uuid: "1c6bbb47e8c24b01a83a6009bc4c8af9" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 46817 } }
I20260619 14:36:11.594058 21976 heartbeater.cc:503] Master 127.2.2.62:36193 was elected leader, sending a full tablet report...
I20260619 14:36:11.594445 22257 ts_tablet_manager.cc:1428] T 28bdf6ab838141fd8a383ec3011681a9 P 1c6bbb47e8c24b01a83a6009bc4c8af9: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260619 14:36:11.594617 22256 raft_consensus.cc:348] T 28bdf6ab838141fd8a383ec3011681a9 P 858c49fb1c43456eb350236121c9b2e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "858c49fb1c43456eb350236121c9b2e2" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 36469 } } peers { permanent_uuid: "c15ac629bf95489f81863e9c1dc817cb" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 46409 } } peers { permanent_uuid: "1c6bbb47e8c24b01a83a6009bc4c8af9" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 46817 } }
I20260619 14:36:11.594838 22256 raft_consensus.cc:374] T 28bdf6ab838141fd8a383ec3011681a9 P 858c49fb1c43456eb350236121c9b2e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:36:11.594894 22256 raft_consensus.cc:729] T 28bdf6ab838141fd8a383ec3011681a9 P 858c49fb1c43456eb350236121c9b2e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 858c49fb1c43456eb350236121c9b2e2, State: Initialized, Role: FOLLOWER
I20260619 14:36:11.595021 22107 heartbeater.cc:503] Master 127.2.2.62:36193 was elected leader, sending a full tablet report...
I20260619 14:36:11.595351 22256 consensus_queue.cc:260] T 28bdf6ab838141fd8a383ec3011681a9 P 858c49fb1c43456eb350236121c9b2e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "858c49fb1c43456eb350236121c9b2e2" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 36469 } } peers { permanent_uuid: "c15ac629bf95489f81863e9c1dc817cb" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 46409 } } peers { permanent_uuid: "1c6bbb47e8c24b01a83a6009bc4c8af9" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 46817 } }
I20260619 14:36:11.595976 22238 heartbeater.cc:503] Master 127.2.2.62:36193 was elected leader, sending a full tablet report...
I20260619 14:36:11.597103 22256 ts_tablet_manager.cc:1428] T 28bdf6ab838141fd8a383ec3011681a9 P 858c49fb1c43456eb350236121c9b2e2: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260619 14:36:11.597743 22255 ts_tablet_manager.cc:1428] T 28bdf6ab838141fd8a383ec3011681a9 P c15ac629bf95489f81863e9c1dc817cb: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
W20260619 14:36:11.608690 21977 tablet.cc:2369] T 28bdf6ab838141fd8a383ec3011681a9 P 1c6bbb47e8c24b01a83a6009bc4c8af9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:36:11.669818 22261 raft_consensus.cc:482] T 28bdf6ab838141fd8a383ec3011681a9 P 1c6bbb47e8c24b01a83a6009bc4c8af9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:36:11.670080 22261 raft_consensus.cc:504] T 28bdf6ab838141fd8a383ec3011681a9 P 1c6bbb47e8c24b01a83a6009bc4c8af9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "858c49fb1c43456eb350236121c9b2e2" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 36469 } } peers { permanent_uuid: "c15ac629bf95489f81863e9c1dc817cb" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 46409 } } peers { permanent_uuid: "1c6bbb47e8c24b01a83a6009bc4c8af9" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 46817 } }
I20260619 14:36:11.671299 22261 leader_election.cc:288] T 28bdf6ab838141fd8a383ec3011681a9 P 1c6bbb47e8c24b01a83a6009bc4c8af9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 858c49fb1c43456eb350236121c9b2e2 (127.2.2.3:36469), c15ac629bf95489f81863e9c1dc817cb (127.2.2.2:46409)
I20260619 14:36:11.677208 22250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:11.671703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:50249 (local address 127.2.2.3:36469)
0619 14:36:11.671953 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:11.671959 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:11.672374 (+ 415us) server_negotiation.cc:407] Connection header received
0619 14:36:11.672461 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:11.672464 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:11.672521 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:11.672615 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:36:11.672946 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:11.673744 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:11.675632 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:11.676288 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:11.676387 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:11.676455 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:11.676980 (+ 525us) server_negotiation.cc:299] Negotiation successful
0619 14:36:11.677087 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260619 14:36:11.677208 22253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:11.672198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.2.1:58069 (local address 127.2.2.2:46409)
0619 14:36:11.672345 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:11.672348 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:11.672468 (+ 120us) server_negotiation.cc:407] Connection header received
0619 14:36:11.672549 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:11.672554 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:11.672615 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:11.672695 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:36:11.672946 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:11.673753 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:11.675632 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:11.676264 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:11.676387 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:11.676475 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:11.676980 (+ 505us) server_negotiation.cc:299] Negotiation successful
0619 14:36:11.677067 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 14:36:11.677281 22265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:11.672056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.2:46409 (local address 127.2.2.1:58069)
0619 14:36:11.672365 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:11.672376 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:36:11.672464 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:36:11.672709 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:36:11.672713 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:36:11.672734 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:36:11.672831 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:11.672837 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:11.673867 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:11.673871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:11.675462 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:11.675474 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:11.676430 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:11.676434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:11.676722 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:11.676850 (+ 128us) client_negotiation.cc:769] Sending connection context
0619 14:36:11.676937 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 14:36:11.677089 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":322,"thread_start_us":119,"threads_started":1}
I20260619 14:36:11.677479 22264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:11.671577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.3:36469 (local address 127.2.2.1:50249)
0619 14:36:11.671975 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:11.671995 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 14:36:11.672171 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:36:11.672628 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:36:11.672630 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:36:11.672643 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:36:11.672812 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:11.672820 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:11.673867 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:11.673871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:11.675462 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:11.675474 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:11.676445 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:11.676448 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:11.676743 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:11.676848 (+ 105us) client_negotiation.cc:769] Sending connection context
0619 14:36:11.676937 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 14:36:11.677090 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":304,"thread_start_us":133,"threads_started":1}
I20260619 14:36:11.677942 22061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28bdf6ab838141fd8a383ec3011681a9" candidate_uuid: "1c6bbb47e8c24b01a83a6009bc4c8af9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c15ac629bf95489f81863e9c1dc817cb" is_pre_election: true
I20260619 14:36:11.678093 22192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28bdf6ab838141fd8a383ec3011681a9" candidate_uuid: "1c6bbb47e8c24b01a83a6009bc4c8af9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "858c49fb1c43456eb350236121c9b2e2" is_pre_election: true
I20260619 14:36:11.678321 22061 raft_consensus.cc:2454] T 28bdf6ab838141fd8a383ec3011681a9 P c15ac629bf95489f81863e9c1dc817cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c6bbb47e8c24b01a83a6009bc4c8af9 in term 0.
I20260619 14:36:11.678442 22192 raft_consensus.cc:2454] T 28bdf6ab838141fd8a383ec3011681a9 P 858c49fb1c43456eb350236121c9b2e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c6bbb47e8c24b01a83a6009bc4c8af9 in term 0.
I20260619 14:36:11.678939 21864 leader_election.cc:302] T 28bdf6ab838141fd8a383ec3011681a9 P 1c6bbb47e8c24b01a83a6009bc4c8af9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c6bbb47e8c24b01a83a6009bc4c8af9, 858c49fb1c43456eb350236121c9b2e2; no voters:
I20260619 14:36:11.679349 22261 raft_consensus.cc:2781] T 28bdf6ab838141fd8a383ec3011681a9 P 1c6bbb47e8c24b01a83a6009bc4c8af9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:36:11.679430 22261 raft_consensus.cc:482] T 28bdf6ab838141fd8a383ec3011681a9 P 1c6bbb47e8c24b01a83a6009bc4c8af9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:36:11.679461 22261 raft_consensus.cc:3037] T 28bdf6ab838141fd8a383ec3011681a9 P 1c6bbb47e8c24b01a83a6009bc4c8af9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:36:11.680766 22261 raft_consensus.cc:504] T 28bdf6ab838141fd8a383ec3011681a9 P 1c6bbb47e8c24b01a83a6009bc4c8af9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "858c49fb1c43456eb350236121c9b2e2" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 36469 } } peers { permanent_uuid: "c15ac629bf95489f81863e9c1dc817cb" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 46409 } } peers { permanent_uuid: "1c6bbb47e8c24b01a83a6009bc4c8af9" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 46817 } }
I20260619 14:36:11.681134 22261 leader_election.cc:288] T 28bdf6ab838141fd8a383ec3011681a9 P 1c6bbb47e8c24b01a83a6009bc4c8af9 [CANDIDATE]: Term 1 election: Requested vote from peers 858c49fb1c43456eb350236121c9b2e2 (127.2.2.3:36469), c15ac629bf95489f81863e9c1dc817cb (127.2.2.2:46409)
I20260619 14:36:11.681627 22192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28bdf6ab838141fd8a383ec3011681a9" candidate_uuid: "1c6bbb47e8c24b01a83a6009bc4c8af9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "858c49fb1c43456eb350236121c9b2e2"
I20260619 14:36:11.681629 22061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28bdf6ab838141fd8a383ec3011681a9" candidate_uuid: "1c6bbb47e8c24b01a83a6009bc4c8af9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c15ac629bf95489f81863e9c1dc817cb"
I20260619 14:36:11.681768 22061 raft_consensus.cc:3037] T 28bdf6ab838141fd8a383ec3011681a9 P c15ac629bf95489f81863e9c1dc817cb [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:36:11.681736 22192 raft_consensus.cc:3037] T 28bdf6ab838141fd8a383ec3011681a9 P 858c49fb1c43456eb350236121c9b2e2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:36:11.682969 22192 raft_consensus.cc:2454] T 28bdf6ab838141fd8a383ec3011681a9 P 858c49fb1c43456eb350236121c9b2e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c6bbb47e8c24b01a83a6009bc4c8af9 in term 1.
I20260619 14:36:11.682984 22061 raft_consensus.cc:2454] T 28bdf6ab838141fd8a383ec3011681a9 P c15ac629bf95489f81863e9c1dc817cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c6bbb47e8c24b01a83a6009bc4c8af9 in term 1.
I20260619 14:36:11.683285 21864 leader_election.cc:302] T 28bdf6ab838141fd8a383ec3011681a9 P 1c6bbb47e8c24b01a83a6009bc4c8af9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c6bbb47e8c24b01a83a6009bc4c8af9, 858c49fb1c43456eb350236121c9b2e2; no voters:
I20260619 14:36:11.683487 22261 raft_consensus.cc:2781] T 28bdf6ab838141fd8a383ec3011681a9 P 1c6bbb47e8c24b01a83a6009bc4c8af9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:36:11.683758 22261 raft_consensus.cc:686] T 28bdf6ab838141fd8a383ec3011681a9 P 1c6bbb47e8c24b01a83a6009bc4c8af9 [term 1 LEADER]: Becoming Leader. State: Replica: 1c6bbb47e8c24b01a83a6009bc4c8af9, State: Running, Role: LEADER
I20260619 14:36:11.684123 22261 consensus_queue.cc:237] T 28bdf6ab838141fd8a383ec3011681a9 P 1c6bbb47e8c24b01a83a6009bc4c8af9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "858c49fb1c43456eb350236121c9b2e2" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 36469 } } peers { permanent_uuid: "c15ac629bf95489f81863e9c1dc817cb" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 46409 } } peers { permanent_uuid: "1c6bbb47e8c24b01a83a6009bc4c8af9" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 46817 } }
I20260619 14:36:11.687208 21788 catalog_manager.cc:5495] T 28bdf6ab838141fd8a383ec3011681a9 P 1c6bbb47e8c24b01a83a6009bc4c8af9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c6bbb47e8c24b01a83a6009bc4c8af9 (127.2.2.1). New cstate: current_term: 1 leader_uuid: "1c6bbb47e8c24b01a83a6009bc4c8af9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "858c49fb1c43456eb350236121c9b2e2" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 36469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c15ac629bf95489f81863e9c1dc817cb" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 46409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c6bbb47e8c24b01a83a6009bc4c8af9" member_type: VOTER last_known_addr { host: "127.2.2.1" port: 46817 } health_report { overall_health: HEALTHY } } }
I20260619 14:36:11.721284 22254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:11.718125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46552 (local address 127.2.2.1:46817)
0619 14:36:11.718276 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:11.718279 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:11.718294 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:36:11.718370 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:11.718372 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:11.718427 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:11.718517 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:36:11.718692 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:11.719213 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:11.720045 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:11.720252 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:11.720330 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:11.720576 (+ 246us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:36:11.720991 (+ 415us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:36:11.721081 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:11.721132 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 14:36:11.721175 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 14:36:11.727432 22192 raft_consensus.cc:1261] T 28bdf6ab838141fd8a383ec3011681a9 P 858c49fb1c43456eb350236121c9b2e2 [term 1 FOLLOWER]: Refusing update from remote peer 1c6bbb47e8c24b01a83a6009bc4c8af9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:36:11.727432 22061 raft_consensus.cc:1261] T 28bdf6ab838141fd8a383ec3011681a9 P c15ac629bf95489f81863e9c1dc817cb [term 1 FOLLOWER]: Refusing update from remote peer 1c6bbb47e8c24b01a83a6009bc4c8af9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:36:11.728215 22266 consensus_queue.cc:1035] T 28bdf6ab838141fd8a383ec3011681a9 P 1c6bbb47e8c24b01a83a6009bc4c8af9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "858c49fb1c43456eb350236121c9b2e2" member_type: VOTER last_known_addr { host: "127.2.2.3" port: 36469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:36:11.728504 22261 consensus_queue.cc:1035] T 28bdf6ab838141fd8a383ec3011681a9 P 1c6bbb47e8c24b01a83a6009bc4c8af9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c15ac629bf95489f81863e9c1dc817cb" member_type: VOTER last_known_addr { host: "127.2.2.2" port: 46409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:36:11.737980 22271 mvcc.cc:205] Tried to move back new op lower bound from 7298579544987070464 to 7298579544821186560. Current Snapshot: MvccSnapshot[applied={T|T < 7298579544987070464}]
I20260619 14:36:11.738202 22270 mvcc.cc:205] Tried to move back new op lower bound from 7298579544987070464 to 7298579544821186560. Current Snapshot: MvccSnapshot[applied={T|T < 7298579544987070464}]
I20260619 14:36:11.744673 21788 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:36:11.744861 21788 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:36:11.747301 21788 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 904d0e5326114ca4877b0bf5790f681f: Sending DeleteTablet for 3 replicas of tablet 28bdf6ab838141fd8a383ec3011681a9
I20260619 14:36:11.748137 22172 tablet_service.cc:1514] Processing DeleteTablet for tablet 28bdf6ab838141fd8a383ec3011681a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:36:11 UTC) from {username='slave'} at 127.0.0.1:51746
I20260619 14:36:11.748129 21910 tablet_service.cc:1514] Processing DeleteTablet for tablet 28bdf6ab838141fd8a383ec3011681a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:36:11 UTC) from {username='slave'} at 127.0.0.1:46538
I20260619 14:36:11.748634 22041 tablet_service.cc:1514] Processing DeleteTablet for tablet 28bdf6ab838141fd8a383ec3011681a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:36:11 UTC) from {username='slave'} at 127.0.0.1:55614
I20260619 14:36:11.749017 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 21848
I20260619 14:36:11.749264 22280 tablet_replica.cc:330] T 28bdf6ab838141fd8a383ec3011681a9 P 858c49fb1c43456eb350236121c9b2e2: stopping tablet replica
I20260619 14:36:11.749558 22280 raft_consensus.cc:2229] T 28bdf6ab838141fd8a383ec3011681a9 P 858c49fb1c43456eb350236121c9b2e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:36:11.749871 22280 raft_consensus.cc:2258] T 28bdf6ab838141fd8a383ec3011681a9 P 858c49fb1c43456eb350236121c9b2e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:36:11.750409 22281 tablet_replica.cc:330] T 28bdf6ab838141fd8a383ec3011681a9 P c15ac629bf95489f81863e9c1dc817cb: stopping tablet replica
I20260619 14:36:11.750702 22281 raft_consensus.cc:2229] T 28bdf6ab838141fd8a383ec3011681a9 P c15ac629bf95489f81863e9c1dc817cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:36:11.751016 22281 raft_consensus.cc:2258] T 28bdf6ab838141fd8a383ec3011681a9 P c15ac629bf95489f81863e9c1dc817cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:36:11.751567 22280 ts_tablet_manager.cc:1905] T 28bdf6ab838141fd8a383ec3011681a9 P 858c49fb1c43456eb350236121c9b2e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:36:11.755187 22280 ts_tablet_manager.cc:1918] T 28bdf6ab838141fd8a383ec3011681a9 P 858c49fb1c43456eb350236121c9b2e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:36:11.755300 22280 log.cc:1196] T 28bdf6ab838141fd8a383ec3011681a9 P 858c49fb1c43456eb350236121c9b2e2: Deleting WAL directory at /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879514993456-2056-0/minicluster-data/ts-2/wal/wals/28bdf6ab838141fd8a383ec3011681a9
I20260619 14:36:11.755290 22281 ts_tablet_manager.cc:1905] T 28bdf6ab838141fd8a383ec3011681a9 P c15ac629bf95489f81863e9c1dc817cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:36:11.755649 22280 ts_tablet_manager.cc:1939] T 28bdf6ab838141fd8a383ec3011681a9 P 858c49fb1c43456eb350236121c9b2e2: Deleting consensus metadata
W20260619 14:36:11.756840 21772 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.2.1:46817 (error 108)
I20260619 14:36:11.757102 21773 catalog_manager.cc:4841] TS 858c49fb1c43456eb350236121c9b2e2 (127.2.2.3:36469): tablet 28bdf6ab838141fd8a383ec3011681a9 (table test-table [id=3c0e8f1865614db49c60710f75430357]) successfully deleted
I20260619 14:36:11.757695 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 21979
I20260619 14:36:11.757980 22252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:11.757673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.1:46817 (local address 127.0.0.1:46562)
0619 14:36:11.757808 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:11.757881 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.1:46817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 14:36:11.762652 21772 catalog_manager.cc:4584] TS 1c6bbb47e8c24b01a83a6009bc4c8af9 (127.2.2.1:46817): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 28bdf6ab838141fd8a383ec3011681a9: Network error: Client connection negotiation failed: client connection to 127.2.2.1:46817: connect: Connection refused (error 111)
I20260619 14:36:11.763991 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 22110
I20260619 14:36:11.764551 22252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:11.764183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.2:46409 (local address 127.0.0.1:55630)
0619 14:36:11.764391 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:11.764448 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.2:46409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260619 14:36:11.769709 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 21757
2026-06-19T14:36:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1148 ms)
[----------] 6 tests from EncryptionPolicyTest (10294 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781879514993456-2056-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:36:11 dist-test-slave-70rm krb5kdc[22285](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:36:11 dist-test-slave-70rm krb5kdc[22285](info): set up 2 sockets
Jun 19 14:36:11 dist-test-slave-70rm krb5kdc[22285](info): commencing operation
krb5kdc: starting...
W20260619 14:36:12.830333 2056 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Jun 19 14:36:12 dist-test-slave-70rm krb5kdc[22285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:36:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:36:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781879514993456-2056-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781879514993456-2056-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:36:12.950814 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781879514993456-2056-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781879514993456-2056-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781879514993456-2056-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38367
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781879514993456-2056-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:36:13.062753 22298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:36:13.063102 22298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:36:13.063186 22298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:36:13.067375 22298 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:36:13.067484 22298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:36:13.067524 22298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:36:13.067584 22298 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:36:13.067637 22298 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:36:13.073237 22298 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781879514993456-2056-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781879514993456-2056-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38367
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781879514993456-2056-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781879514993456-2056-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:36:13.074481 22298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:36:13.075610 22298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:36:13.084403 22303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:13.084403 22304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:13.084508 22306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:13.084590 22298 server_base.cc:1034] running on GCE node
I20260619 14:36:13.085332 22298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:36:13.086630 22298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:36:13.087828 22298 hybrid_clock.cc:648] HybridClock initialized: now 1781879773087795 us; error 52 us; skew 500 ppm
Jun 19 14:36:13 dist-test-slave-70rm krb5kdc[22285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879773, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:36:13.090673 22298 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:36:13.091688 22298 webserver.cc:460] Webserver started at http://127.0.0.1:44467/ using document root <none> and password file <none>
I20260619 14:36:13.092293 22298 fs_manager.cc:359] Metadata directory not provided
I20260619 14:36:13.092391 22298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:36:13.092643 22298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:36:13.094991 22298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781879514993456-2056-0/minicluster-data/master-0/data/instance:
uuid: "37a80e4826534539990be17cdd6205fb"
format_stamp: "Formatted at 2026-06-19 14:36:13 on dist-test-slave-70rm"
server_key: "c35b065fe855d1a21a798a5b1ad227f6"
server_key_iv: "09802f5fa7aef5ad494958f06cbb15e2"
server_key_version: "encryptionkey@0"
I20260619 14:36:13.095598 22298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781879514993456-2056-0/minicluster-data/master-0/wal/instance:
uuid: "37a80e4826534539990be17cdd6205fb"
format_stamp: "Formatted at 2026-06-19 14:36:13 on dist-test-slave-70rm"
server_key: "c35b065fe855d1a21a798a5b1ad227f6"
server_key_iv: "09802f5fa7aef5ad494958f06cbb15e2"
server_key_version: "encryptionkey@0"
I20260619 14:36:13.099897 22298 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:36:13.102792 22313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:13.104164 22298 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:36:13.104334 22298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781879514993456-2056-0/minicluster-data/master-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781879514993456-2056-0/minicluster-data/master-0/wal
uuid: "37a80e4826534539990be17cdd6205fb"
format_stamp: "Formatted at 2026-06-19 14:36:13 on dist-test-slave-70rm"
server_key: "c35b065fe855d1a21a798a5b1ad227f6"
server_key_iv: "09802f5fa7aef5ad494958f06cbb15e2"
server_key_version: "encryptionkey@0"
I20260619 14:36:13.104489 22298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781879514993456-2056-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781879514993456-2056-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781879514993456-2056-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:36:13.129835 22298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:36:13.133409 22298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:36:13.133662 22298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:36:13.140821 22298 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38367
I20260619 14:36:13.140830 22365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38367 every 8 connection(s)
I20260619 14:36:13.141805 22298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
I20260619 14:36:13.144641 22366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:36:13.147349 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 22298
I20260619 14:36:13.147507 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781879514993456-2056-0/minicluster-data/master-0/wal/instance
I20260619 14:36:13.147801 2056 external_mini_cluster.cc:1428] Setting key e9712c75c27ffb883053a07130f80ddc
I20260619 14:36:13.151213 22366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37a80e4826534539990be17cdd6205fb: Bootstrap starting.
I20260619 14:36:13.153475 22366 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37a80e4826534539990be17cdd6205fb: Neither blocks nor log segments found. Creating new log.
I20260619 14:36:13.154320 22366 log.cc:824] T 00000000000000000000000000000000 P 37a80e4826534539990be17cdd6205fb: Log is configured to *not* fsync() on all Append() calls
I20260619 14:36:13.156275 22366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37a80e4826534539990be17cdd6205fb: No bootstrap required, opened a new log
Jun 19 14:36:13 dist-test-slave-70rm krb5kdc[22285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 14:36:13.159852 22366 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37a80e4826534539990be17cdd6205fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37a80e4826534539990be17cdd6205fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38367 } }
I20260619 14:36:13.160131 22366 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37a80e4826534539990be17cdd6205fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:36:13.160215 22366 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37a80e4826534539990be17cdd6205fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37a80e4826534539990be17cdd6205fb, State: Initialized, Role: FOLLOWER
I20260619 14:36:13.160748 22366 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37a80e4826534539990be17cdd6205fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37a80e4826534539990be17cdd6205fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38367 } }
I20260619 14:36:13.160909 22366 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37a80e4826534539990be17cdd6205fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:36:13.160997 22366 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37a80e4826534539990be17cdd6205fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:36:13.161100 22366 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37a80e4826534539990be17cdd6205fb [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:36:13.162071 22366 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37a80e4826534539990be17cdd6205fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37a80e4826534539990be17cdd6205fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38367 } }
I20260619 14:36:13.162524 22366 leader_election.cc:302] T 00000000000000000000000000000000 P 37a80e4826534539990be17cdd6205fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37a80e4826534539990be17cdd6205fb; no voters:
I20260619 14:36:13.162917 22366 leader_election.cc:288] T 00000000000000000000000000000000 P 37a80e4826534539990be17cdd6205fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:36:13.162999 22371 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37a80e4826534539990be17cdd6205fb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:36:13.163224 22371 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37a80e4826534539990be17cdd6205fb [term 1 LEADER]: Becoming Leader. State: Replica: 37a80e4826534539990be17cdd6205fb, State: Running, Role: LEADER
I20260619 14:36:13.163591 22371 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37a80e4826534539990be17cdd6205fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37a80e4826534539990be17cdd6205fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38367 } }
I20260619 14:36:13.163733 22369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:13.149319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51475 (local address 127.0.0.1:38367)
0619 14:36:13.149795 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:13.149812 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:13.149849 (+ 37us) server_negotiation.cc:407] Connection header received
0619 14:36:13.150568 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:13.150587 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:13.150948 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:13.151292 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:13.152105 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:13.153098 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:13.154044 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:13.154654 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:13.154831 (+ 177us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:13.157417 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:13.157422 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:13.157436 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:36:13.157464 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:13.160129 (+ 2665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:13.160768 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:13.160774 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:13.160778 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:13.160921 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:13.161324 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:13.161327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:13.161329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:13.161751 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:13.161992 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:13.162351 (+ 359us) server_negotiation.cc:299] Negotiation successful
0619 14:36:13.162590 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":145,"threads_started":1}
I20260619 14:36:13.164288 22366 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37a80e4826534539990be17cdd6205fb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:36:13.165326 22372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37a80e4826534539990be17cdd6205fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37a80e4826534539990be17cdd6205fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37a80e4826534539990be17cdd6205fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38367 } } }
I20260619 14:36:13.165447 22372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37a80e4826534539990be17cdd6205fb [sys.catalog]: This master's current role is: LEADER
I20260619 14:36:13.165848 22380 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:36:13.165846 22373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37a80e4826534539990be17cdd6205fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37a80e4826534539990be17cdd6205fb. Latest consensus state: current_term: 1 leader_uuid: "37a80e4826534539990be17cdd6205fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37a80e4826534539990be17cdd6205fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38367 } } }
I20260619 14:36:13.166000 22373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37a80e4826534539990be17cdd6205fb [sys.catalog]: This master's current role is: LEADER
I20260619 14:36:13.167158 2056 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:36:13.169607 22380 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:36:13.176347 22380 catalog_manager.cc:1347] Generated new cluster ID: c1c7aa07cee0408aab7ffba6fb839ca2
I20260619 14:36:13.176474 22380 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:36:13.178588 22369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:13.169444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37653 (local address 127.0.0.1:38367)
0619 14:36:13.169557 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:13.169561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:13.169743 (+ 182us) server_negotiation.cc:407] Connection header received
0619 14:36:13.170109 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:13.170112 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:13.170197 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:13.170259 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:13.171158 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:13.171709 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:13.172540 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:13.172794 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:13.172910 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:13.174628 (+ 1718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:13.174632 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:13.174635 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:36:13.174664 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:13.176463 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:13.177151 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:13.177155 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:13.177157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:13.177263 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:13.177635 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:13.177640 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:13.177643 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:13.177871 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:13.178037 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:13.178318 (+ 281us) server_negotiation.cc:299] Negotiation successful
0619 14:36:13.178421 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 14:36:13.204044 22380 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:36:13.205046 22380 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:36:13.211777 22380 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37a80e4826534539990be17cdd6205fb: Generated new TSK 0
I20260619 14:36:13.212550 22380 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:36:13.221907 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 22298
W20260619 14:36:13.227272 22283 connection.cc:535] client connection to 127.0.0.1:38367 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38367): Connection reset by peer (error 104)
2026-06-19T14:36:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1450 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781879514993456-2056-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:36:13 dist-test-slave-70rm krb5kdc[22397](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:36:13 dist-test-slave-70rm krb5kdc[22397](info): set up 2 sockets
Jun 19 14:36:13 dist-test-slave-70rm krb5kdc[22397](info): commencing operation
krb5kdc: starting...
W20260619 14:36:14.254462 2056 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Jun 19 14:36:14 dist-test-slave-70rm krb5kdc[22397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:36:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:36:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781879514993456-2056-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781879514993456-2056-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:36:14.376884 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781879514993456-2056-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781879514993456-2056-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781879514993456-2056-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37781
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781879514993456-2056-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:36:14.487062 22410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:36:14.487339 22410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:36:14.487432 22410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:36:14.491508 22410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:36:14.491580 22410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:36:14.491601 22410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:36:14.491668 22410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:36:14.491698 22410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:36:14.497102 22410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781879514993456-2056-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781879514993456-2056-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37781
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781879514993456-2056-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781879514993456-2056-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:36:14.498229 22410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:36:14.499374 22410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:36:14.507997 22418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:14.508020 22415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:14.508128 22410 server_base.cc:1034] running on GCE node
W20260619 14:36:14.508020 22416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:14.508955 22410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:36:14.510165 22410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:36:14.511395 22410 hybrid_clock.cc:648] HybridClock initialized: now 1781879774511376 us; error 35 us; skew 500 ppm
Jun 19 14:36:14 dist-test-slave-70rm krb5kdc[22397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:36:14.514358 22410 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:36:14.515230 22410 webserver.cc:460] Webserver started at http://127.0.0.1:34255/ using document root <none> and password file <none>
I20260619 14:36:14.515774 22410 fs_manager.cc:359] Metadata directory not provided
I20260619 14:36:14.515833 22410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:36:14.515997 22410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:36:14.517998 22410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781879514993456-2056-0/minicluster-data/master-0/data/instance:
uuid: "5f9f164772e54e9889388e441d99645d"
format_stamp: "Formatted at 2026-06-19 14:36:14 on dist-test-slave-70rm"
server_key: "38595316f5171c552ce32454e0ccfce1"
server_key_iv: "0747507d56d4095d329c840e16d21c41"
server_key_version: "encryptionkey@0"
I20260619 14:36:14.518600 22410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781879514993456-2056-0/minicluster-data/master-0/wal/instance:
uuid: "5f9f164772e54e9889388e441d99645d"
format_stamp: "Formatted at 2026-06-19 14:36:14 on dist-test-slave-70rm"
server_key: "38595316f5171c552ce32454e0ccfce1"
server_key_iv: "0747507d56d4095d329c840e16d21c41"
server_key_version: "encryptionkey@0"
I20260619 14:36:14.522289 22410 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:36:14.524679 22425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:14.525897 22410 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:36:14.525996 22410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781879514993456-2056-0/minicluster-data/master-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781879514993456-2056-0/minicluster-data/master-0/wal
uuid: "5f9f164772e54e9889388e441d99645d"
format_stamp: "Formatted at 2026-06-19 14:36:14 on dist-test-slave-70rm"
server_key: "38595316f5171c552ce32454e0ccfce1"
server_key_iv: "0747507d56d4095d329c840e16d21c41"
server_key_version: "encryptionkey@0"
I20260619 14:36:14.526085 22410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781879514993456-2056-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781879514993456-2056-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781879514993456-2056-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:36:14.541168 22410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:36:14.544413 22410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:36:14.544579 22410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:36:14.551594 22410 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37781
I20260619 14:36:14.551712 22477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37781 every 8 connection(s)
I20260619 14:36:14.552620 22410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
I20260619 14:36:14.553145 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 22410
I20260619 14:36:14.553306 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781879514993456-2056-0/minicluster-data/master-0/wal/instance
I20260619 14:36:14.553550 2056 external_mini_cluster.cc:1428] Setting key 1273793cdf3d367f06c90e7ecae6d6cb
I20260619 14:36:14.555966 22478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:36:14.563117 22478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f9f164772e54e9889388e441d99645d: Bootstrap starting.
I20260619 14:36:14.563864 22481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:14.554963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58937 (local address 127.0.0.1:37781)
0619 14:36:14.555401 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:14.555421 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:14.555463 (+ 42us) server_negotiation.cc:407] Connection header received
0619 14:36:14.556325 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:14.556348 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:14.556775 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:14.557135 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:14.557549 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:14.559043 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:14.559937 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:14.560630 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:14.560800 (+ 170us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:14.561707 (+ 907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:14.561718 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:14.561738 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:36:14.562324 (+ 586us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:14.562397 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:36:14.562426 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:14.562560 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:14.562747 (+ 187us) server_negotiation.cc:299] Negotiation successful
0619 14:36:14.562890 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":115,"threads_started":1}
I20260619 14:36:14.565768 22478 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f9f164772e54e9889388e441d99645d: Neither blocks nor log segments found. Creating new log.
I20260619 14:36:14.566673 22478 log.cc:824] T 00000000000000000000000000000000 P 5f9f164772e54e9889388e441d99645d: Log is configured to *not* fsync() on all Append() calls
I20260619 14:36:14.566916 2056 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:36:14.568773 22478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f9f164772e54e9889388e441d99645d: No bootstrap required, opened a new log
I20260619 14:36:14.573036 22481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:14.569043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48665 (local address 127.0.0.1:37781)
0619 14:36:14.569178 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:14.569182 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:14.569336 (+ 154us) server_negotiation.cc:407] Connection header received
0619 14:36:14.569414 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:14.569417 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:14.569461 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:14.569541 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:14.569764 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:14.570564 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:14.571362 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:14.571687 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:14.571799 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:14.572624 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:14.572628 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:14.572631 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:36:14.572645 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:14.572670 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:36:14.572683 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:14.572776 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:14.572871 (+ 95us) server_negotiation.cc:299] Negotiation successful
0619 14:36:14.572907 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 14:36:14.573220 22478 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f9f164772e54e9889388e441d99645d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f9f164772e54e9889388e441d99645d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37781 } }
I20260619 14:36:14.573510 22478 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f9f164772e54e9889388e441d99645d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:36:14.573594 22478 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f9f164772e54e9889388e441d99645d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f9f164772e54e9889388e441d99645d, State: Initialized, Role: FOLLOWER
I20260619 14:36:14.574106 22478 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f9f164772e54e9889388e441d99645d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f9f164772e54e9889388e441d99645d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37781 } }
I20260619 14:36:14.574313 22478 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f9f164772e54e9889388e441d99645d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:36:14.574379 22478 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f9f164772e54e9889388e441d99645d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:36:14.574477 22478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f9f164772e54e9889388e441d99645d [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:36:14.575425 22478 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f9f164772e54e9889388e441d99645d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f9f164772e54e9889388e441d99645d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37781 } }
I20260619 14:36:14.575803 22478 leader_election.cc:302] T 00000000000000000000000000000000 P 5f9f164772e54e9889388e441d99645d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f9f164772e54e9889388e441d99645d; no voters:
I20260619 14:36:14.576134 22478 leader_election.cc:288] T 00000000000000000000000000000000 P 5f9f164772e54e9889388e441d99645d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:36:14.576287 22488 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f9f164772e54e9889388e441d99645d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:36:14.576539 22488 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f9f164772e54e9889388e441d99645d [term 1 LEADER]: Becoming Leader. State: Replica: 5f9f164772e54e9889388e441d99645d, State: Running, Role: LEADER
I20260619 14:36:14.576874 22488 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f9f164772e54e9889388e441d99645d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f9f164772e54e9889388e441d99645d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37781 } }
I20260619 14:36:14.577401 22478 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f9f164772e54e9889388e441d99645d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:36:14.578563 22490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f9f164772e54e9889388e441d99645d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f9f164772e54e9889388e441d99645d. Latest consensus state: current_term: 1 leader_uuid: "5f9f164772e54e9889388e441d99645d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f9f164772e54e9889388e441d99645d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37781 } } }
I20260619 14:36:14.578701 22490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f9f164772e54e9889388e441d99645d [sys.catalog]: This master's current role is: LEADER
I20260619 14:36:14.579064 22497 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:36:14.579046 22489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f9f164772e54e9889388e441d99645d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f9f164772e54e9889388e441d99645d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f9f164772e54e9889388e441d99645d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37781 } } }
I20260619 14:36:14.579262 22489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f9f164772e54e9889388e441d99645d [sys.catalog]: This master's current role is: LEADER
I20260619 14:36:14.582412 22497 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:36:14.588375 22497 catalog_manager.cc:1347] Generated new cluster ID: 5c44e86e653b4e1a8d84c933eed67425
I20260619 14:36:14.588459 22497 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:36:14.607669 22497 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:36:14.608676 22497 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:36:14.617437 22497 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f9f164772e54e9889388e441d99645d: Generated new TSK 0
I20260619 14:36:14.618271 22497 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:36:14.629266 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 22410
W20260619 14:36:14.634714 22395 connection.cc:535] client connection to 127.0.0.1:37781 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37781): Connection reset by peer (error 104)
2026-06-19T14:36:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1407 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781879514993456-2056-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:36:14 dist-test-slave-70rm krb5kdc[22509](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:36:14 dist-test-slave-70rm krb5kdc[22509](info): set up 2 sockets
Jun 19 14:36:14 dist-test-slave-70rm krb5kdc[22509](info): commencing operation
krb5kdc: starting...
W20260619 14:36:15.674393 2056 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Jun 19 14:36:15 dist-test-slave-70rm krb5kdc[22509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:36:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:36:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781879514993456-2056-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781879514993456-2056-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:36:15.797022 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781879514993456-2056-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781879514993456-2056-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781879514993456-2056-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46383
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781879514993456-2056-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:36:15.907356 22522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:36:15.907711 22522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:36:15.907795 22522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:36:15.911922 22522 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:36:15.912025 22522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:36:15.912063 22522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:36:15.912104 22522 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:36:15.912160 22522 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:36:15.917667 22522 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781879514993456-2056-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781879514993456-2056-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46383
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781879514993456-2056-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22522
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781879514993456-2056-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:36:15.918862 22522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:36:15.919957 22522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:36:15.928496 22527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:15.928496 22528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:15.928565 22522 server_base.cc:1034] running on GCE node
W20260619 14:36:15.928496 22530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:15.929371 22522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:36:15.930532 22522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:36:15.931715 22522 hybrid_clock.cc:648] HybridClock initialized: now 1781879775931686 us; error 44 us; skew 500 ppm
Jun 19 14:36:15 dist-test-slave-70rm krb5kdc[22509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879775, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:36:15.934506 22522 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:36:15.935415 22522 webserver.cc:460] Webserver started at http://127.0.0.1:39361/ using document root <none> and password file <none>
I20260619 14:36:15.935971 22522 fs_manager.cc:359] Metadata directory not provided
I20260619 14:36:15.936058 22522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:36:15.936296 22522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:36:15.938416 22522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781879514993456-2056-0/minicluster-data/master-0/data/instance:
uuid: "24c6aa6f49fe4b8cac4826eadb5ac133"
format_stamp: "Formatted at 2026-06-19 14:36:15 on dist-test-slave-70rm"
server_key: "5150314440432b22f5529f5a7b74e5d7"
server_key_iv: "da69c4932c548e007e5fb216203db6ff"
server_key_version: "encryptionkey@0"
I20260619 14:36:15.938971 22522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781879514993456-2056-0/minicluster-data/master-0/wal/instance:
uuid: "24c6aa6f49fe4b8cac4826eadb5ac133"
format_stamp: "Formatted at 2026-06-19 14:36:15 on dist-test-slave-70rm"
server_key: "5150314440432b22f5529f5a7b74e5d7"
server_key_iv: "da69c4932c548e007e5fb216203db6ff"
server_key_version: "encryptionkey@0"
I20260619 14:36:15.942720 22522 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 14:36:15.945182 22537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:15.946493 22522 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:36:15.946642 22522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781879514993456-2056-0/minicluster-data/master-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781879514993456-2056-0/minicluster-data/master-0/wal
uuid: "24c6aa6f49fe4b8cac4826eadb5ac133"
format_stamp: "Formatted at 2026-06-19 14:36:15 on dist-test-slave-70rm"
server_key: "5150314440432b22f5529f5a7b74e5d7"
server_key_iv: "da69c4932c548e007e5fb216203db6ff"
server_key_version: "encryptionkey@0"
I20260619 14:36:15.946767 22522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781879514993456-2056-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781879514993456-2056-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781879514993456-2056-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:36:15.951560 22522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:36:15.954932 22522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:36:15.955139 22522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:36:15.962236 22522 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46383
I20260619 14:36:15.962255 22589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46383 every 8 connection(s)
I20260619 14:36:15.963172 22522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
I20260619 14:36:15.965915 22590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:36:15.971503 22590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24c6aa6f49fe4b8cac4826eadb5ac133: Bootstrap starting.
I20260619 14:36:15.973444 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 22522
I20260619 14:36:15.973603 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781879514993456-2056-0/minicluster-data/master-0/wal/instance
I20260619 14:36:15.973877 2056 external_mini_cluster.cc:1428] Setting key 7b7a1b6e6a690108df78b570515ecffd
I20260619 14:36:15.974359 22590 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24c6aa6f49fe4b8cac4826eadb5ac133: Neither blocks nor log segments found. Creating new log.
I20260619 14:36:15.975234 22590 log.cc:824] T 00000000000000000000000000000000 P 24c6aa6f49fe4b8cac4826eadb5ac133: Log is configured to *not* fsync() on all Append() calls
I20260619 14:36:15.977357 22590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24c6aa6f49fe4b8cac4826eadb5ac133: No bootstrap required, opened a new log
I20260619 14:36:15.979169 22593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:15.975091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58873 (local address 127.0.0.1:46383)
0619 14:36:15.975526 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:15.975544 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:15.975582 (+ 38us) server_negotiation.cc:407] Connection header received
0619 14:36:15.976439 (+ 857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:15.976459 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:15.976745 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:15.977056 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:15.977912 (+ 856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:15.977917 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:15.977929 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:36:15.977939 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:15.977989 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:36:15.978008 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:15.978101 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:15.978259 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 14:36:15.978430 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":146,"threads_started":1}
I20260619 14:36:15.981667 22590 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24c6aa6f49fe4b8cac4826eadb5ac133 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24c6aa6f49fe4b8cac4826eadb5ac133" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46383 } }
I20260619 14:36:15.981954 2056 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:36:15.981967 22590 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24c6aa6f49fe4b8cac4826eadb5ac133 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:36:15.982030 22590 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24c6aa6f49fe4b8cac4826eadb5ac133 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24c6aa6f49fe4b8cac4826eadb5ac133, State: Initialized, Role: FOLLOWER
I20260619 14:36:15.982497 22590 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24c6aa6f49fe4b8cac4826eadb5ac133 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24c6aa6f49fe4b8cac4826eadb5ac133" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46383 } }
I20260619 14:36:15.982656 22590 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24c6aa6f49fe4b8cac4826eadb5ac133 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:36:15.982725 22590 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24c6aa6f49fe4b8cac4826eadb5ac133 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:36:15.982822 22590 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24c6aa6f49fe4b8cac4826eadb5ac133 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:36:15.983850 22590 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24c6aa6f49fe4b8cac4826eadb5ac133 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24c6aa6f49fe4b8cac4826eadb5ac133" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46383 } }
I20260619 14:36:15.984236 22590 leader_election.cc:302] T 00000000000000000000000000000000 P 24c6aa6f49fe4b8cac4826eadb5ac133 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24c6aa6f49fe4b8cac4826eadb5ac133; no voters:
I20260619 14:36:15.984598 22590 leader_election.cc:288] T 00000000000000000000000000000000 P 24c6aa6f49fe4b8cac4826eadb5ac133 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:36:15.984661 22600 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24c6aa6f49fe4b8cac4826eadb5ac133 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:36:15.984879 22600 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24c6aa6f49fe4b8cac4826eadb5ac133 [term 1 LEADER]: Becoming Leader. State: Replica: 24c6aa6f49fe4b8cac4826eadb5ac133, State: Running, Role: LEADER
I20260619 14:36:15.985224 22600 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24c6aa6f49fe4b8cac4826eadb5ac133 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24c6aa6f49fe4b8cac4826eadb5ac133" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46383 } }
I20260619 14:36:15.985860 22593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:15.984083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35593 (local address 127.0.0.1:46383)
0619 14:36:15.984209 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:15.984214 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:15.984379 (+ 165us) server_negotiation.cc:407] Connection header received
0619 14:36:15.984542 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:15.984545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:15.984607 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:15.984701 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:15.985461 (+ 760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:15.985464 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:15.985466 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:36:15.985477 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:15.985501 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:36:15.985511 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:15.985590 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:15.985694 (+ 104us) server_negotiation.cc:299] Negotiation successful
0619 14:36:15.985734 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 14:36:15.985896 22590 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24c6aa6f49fe4b8cac4826eadb5ac133 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:36:15.987252 22602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24c6aa6f49fe4b8cac4826eadb5ac133 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24c6aa6f49fe4b8cac4826eadb5ac133. Latest consensus state: current_term: 1 leader_uuid: "24c6aa6f49fe4b8cac4826eadb5ac133" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24c6aa6f49fe4b8cac4826eadb5ac133" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46383 } } }
I20260619 14:36:15.987378 22602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24c6aa6f49fe4b8cac4826eadb5ac133 [sys.catalog]: This master's current role is: LEADER
I20260619 14:36:15.987744 22611 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:36:15.987733 22601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24c6aa6f49fe4b8cac4826eadb5ac133 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24c6aa6f49fe4b8cac4826eadb5ac133" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24c6aa6f49fe4b8cac4826eadb5ac133" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46383 } } }
I20260619 14:36:15.988131 22601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24c6aa6f49fe4b8cac4826eadb5ac133 [sys.catalog]: This master's current role is: LEADER
I20260619 14:36:15.990617 22611 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:36:15.996160 22611 catalog_manager.cc:1347] Generated new cluster ID: 6fb51b18519347a0b18e509f6e1cf676
I20260619 14:36:15.996256 22611 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:36:16.010279 22611 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:36:16.010617 22611 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:36:16.017823 22611 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24c6aa6f49fe4b8cac4826eadb5ac133: Generated new TSK 0
I20260619 14:36:16.018718 22611 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:36:16.022181 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 22522
2026-06-19T14:36:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1391 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781879514993456-2056-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:36:16 dist-test-slave-70rm krb5kdc[22621](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:36:16 dist-test-slave-70rm krb5kdc[22621](info): set up 2 sockets
Jun 19 14:36:16 dist-test-slave-70rm krb5kdc[22621](info): commencing operation
krb5kdc: starting...
W20260619 14:36:17.070891 2056 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Jun 19 14:36:17 dist-test-slave-70rm krb5kdc[22621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:36:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:36:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781879514993456-2056-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781879514993456-2056-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:36:17.194792 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781879514993456-2056-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781879514993456-2056-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781879514993456-2056-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42435
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781879514993456-2056-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42435
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:36:17.304592 22634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:36:17.304890 22634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:36:17.304982 22634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:36:17.309198 22634 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:36:17.309269 22634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:36:17.309289 22634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:36:17.309305 22634 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:36:17.309321 22634 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:36:17.314648 22634 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781879514993456-2056-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781879514993456-2056-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42435
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781879514993456-2056-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781879514993456-2056-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:36:17.315663 22634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:36:17.316653 22634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:36:17.325166 22639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:17.325188 22642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:17.325227 22634 server_base.cc:1034] running on GCE node
W20260619 14:36:17.325173 22640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:17.325992 22634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:36:17.327196 22634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:36:17.328366 22634 hybrid_clock.cc:648] HybridClock initialized: now 1781879777328350 us; error 29 us; skew 500 ppm
Jun 19 14:36:17 dist-test-slave-70rm krb5kdc[22621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879777, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:36:17.336728 22634 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:36:17.337626 22634 webserver.cc:460] Webserver started at http://127.0.0.1:43183/ using document root <none> and password file <none>
I20260619 14:36:17.338243 22634 fs_manager.cc:359] Metadata directory not provided
I20260619 14:36:17.338312 22634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:36:17.338485 22634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:36:17.340516 22634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781879514993456-2056-0/minicluster-data/master-0/data/instance:
uuid: "8923847190cd4db9a1b15486dde91404"
format_stamp: "Formatted at 2026-06-19 14:36:17 on dist-test-slave-70rm"
server_key: "cc30b56fc65729f664df469bcd66c5cf"
server_key_iv: "4de3a7eec89ae8073a2d74e019131246"
server_key_version: "encryptionkey@0"
I20260619 14:36:17.341041 22634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781879514993456-2056-0/minicluster-data/master-0/wal/instance:
uuid: "8923847190cd4db9a1b15486dde91404"
format_stamp: "Formatted at 2026-06-19 14:36:17 on dist-test-slave-70rm"
server_key: "cc30b56fc65729f664df469bcd66c5cf"
server_key_iv: "4de3a7eec89ae8073a2d74e019131246"
server_key_version: "encryptionkey@0"
I20260619 14:36:17.344689 22634 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 14:36:17.347093 22649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:17.348305 22634 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:36:17.348439 22634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781879514993456-2056-0/minicluster-data/master-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781879514993456-2056-0/minicluster-data/master-0/wal
uuid: "8923847190cd4db9a1b15486dde91404"
format_stamp: "Formatted at 2026-06-19 14:36:17 on dist-test-slave-70rm"
server_key: "cc30b56fc65729f664df469bcd66c5cf"
server_key_iv: "4de3a7eec89ae8073a2d74e019131246"
server_key_version: "encryptionkey@0"
I20260619 14:36:17.348547 22634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781879514993456-2056-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781879514993456-2056-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781879514993456-2056-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:36:17.365808 22634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:36:17.368844 22634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:36:17.369064 22634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:36:17.376231 22634 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42435
I20260619 14:36:17.376257 22701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42435 every 8 connection(s)
I20260619 14:36:17.377205 22634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
I20260619 14:36:17.380025 22702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:36:17.381049 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 22634
I20260619 14:36:17.381201 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781879514993456-2056-0/minicluster-data/master-0/wal/instance
I20260619 14:36:17.381527 2056 external_mini_cluster.cc:1428] Setting key e61a9f45ec7d03dc4ef56cb1e74cefe5
I20260619 14:36:17.386211 22702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8923847190cd4db9a1b15486dde91404: Bootstrap starting.
I20260619 14:36:17.388458 22702 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8923847190cd4db9a1b15486dde91404: Neither blocks nor log segments found. Creating new log.
I20260619 14:36:17.389386 22702 log.cc:824] T 00000000000000000000000000000000 P 8923847190cd4db9a1b15486dde91404: Log is configured to *not* fsync() on all Append() calls
Jun 19 14:36:17 dist-test-slave-70rm krb5kdc[22621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 14:36:17.391793 22702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8923847190cd4db9a1b15486dde91404: No bootstrap required, opened a new log
I20260619 14:36:17.396927 22705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:17.382977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32913 (local address 127.0.0.1:42435)
0619 14:36:17.383411 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:17.383427 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:17.383465 (+ 38us) server_negotiation.cc:407] Connection header received
0619 14:36:17.384125 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:17.384143 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:17.384516 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:17.384873 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:17.385753 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:17.386894 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:17.387818 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:17.388319 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:17.388517 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:17.391486 (+ 2969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:17.391493 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:17.391510 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:36:17.391544 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:17.393541 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:17.393989 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:17.393996 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:17.394006 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:17.394195 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:17.394522 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:17.394527 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:17.394529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:17.395081 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:17.395289 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:17.395566 (+ 277us) server_negotiation.cc:299] Negotiation successful
0619 14:36:17.395891 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":127,"threads_started":1}
I20260619 14:36:17.396796 22702 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8923847190cd4db9a1b15486dde91404 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8923847190cd4db9a1b15486dde91404" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42435 } }
I20260619 14:36:17.397070 22702 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8923847190cd4db9a1b15486dde91404 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:36:17.397121 22702 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8923847190cd4db9a1b15486dde91404 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8923847190cd4db9a1b15486dde91404, State: Initialized, Role: FOLLOWER
I20260619 14:36:17.397614 22702 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8923847190cd4db9a1b15486dde91404 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8923847190cd4db9a1b15486dde91404" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42435 } }
I20260619 14:36:17.397776 22702 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8923847190cd4db9a1b15486dde91404 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:36:17.397830 22702 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8923847190cd4db9a1b15486dde91404 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:36:17.397972 22702 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8923847190cd4db9a1b15486dde91404 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:36:17.399060 22702 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8923847190cd4db9a1b15486dde91404 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8923847190cd4db9a1b15486dde91404" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42435 } }
I20260619 14:36:17.399421 22702 leader_election.cc:302] T 00000000000000000000000000000000 P 8923847190cd4db9a1b15486dde91404 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8923847190cd4db9a1b15486dde91404; no voters:
I20260619 14:36:17.399844 22702 leader_election.cc:288] T 00000000000000000000000000000000 P 8923847190cd4db9a1b15486dde91404 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:36:17.399926 22707 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8923847190cd4db9a1b15486dde91404 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:36:17.400099 2056 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:36:17.400133 22707 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8923847190cd4db9a1b15486dde91404 [term 1 LEADER]: Becoming Leader. State: Replica: 8923847190cd4db9a1b15486dde91404, State: Running, Role: LEADER
I20260619 14:36:17.400429 22707 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8923847190cd4db9a1b15486dde91404 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8923847190cd4db9a1b15486dde91404" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42435 } }
I20260619 14:36:17.401283 22702 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8923847190cd4db9a1b15486dde91404 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:36:17.403913 22709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8923847190cd4db9a1b15486dde91404 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8923847190cd4db9a1b15486dde91404" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8923847190cd4db9a1b15486dde91404" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42435 } } }
I20260619 14:36:17.404047 22709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8923847190cd4db9a1b15486dde91404 [sys.catalog]: This master's current role is: LEADER
I20260619 14:36:17.404387 22708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8923847190cd4db9a1b15486dde91404 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8923847190cd4db9a1b15486dde91404. Latest consensus state: current_term: 1 leader_uuid: "8923847190cd4db9a1b15486dde91404" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8923847190cd4db9a1b15486dde91404" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42435 } } }
I20260619 14:36:17.404686 22708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8923847190cd4db9a1b15486dde91404 [sys.catalog]: This master's current role is: LEADER
W20260619 14:36:17.406932 22727 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8923847190cd4db9a1b15486dde91404: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:36:17.407018 22727 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:36:17.407099 22728 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:36:17.407925 22728 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:36:17.413796 22705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:17.402534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34699 (local address 127.0.0.1:42435)
0619 14:36:17.402642 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:17.402647 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:17.404733 (+ 2086us) server_negotiation.cc:407] Connection header received
0619 14:36:17.404871 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:17.404873 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:17.404913 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:17.404985 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:17.405856 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:17.406710 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:17.407808 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:17.408179 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:17.408295 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:17.410001 (+ 1706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:17.410007 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:17.410011 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:36:17.410043 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:17.411887 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:17.412423 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:17.412428 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:17.412430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:17.412519 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:17.412829 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:17.412833 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:17.412835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:17.413077 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:17.413217 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:17.413493 (+ 276us) server_negotiation.cc:299] Negotiation successful
0619 14:36:17.413627 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 14:36:17.414355 22728 catalog_manager.cc:1347] Generated new cluster ID: e06107e99d7943ffae22ffaf22bb3244
I20260619 14:36:17.414410 22728 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:36:17.435832 22728 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:36:17.437029 22728 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:36:17.441365 22728 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8923847190cd4db9a1b15486dde91404: Generated new TSK 0
I20260619 14:36:17.442036 22728 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:36:17.446880 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 22634
2026-06-19T14:36:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1425 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781879514993456-2056-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:36:17 dist-test-slave-70rm krb5kdc[22733](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:36:17 dist-test-slave-70rm krb5kdc[22733](info): set up 2 sockets
Jun 19 14:36:17 dist-test-slave-70rm krb5kdc[22733](info): commencing operation
krb5kdc: starting...
W20260619 14:36:18.482333 2056 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Jun 19 14:36:18 dist-test-slave-70rm krb5kdc[22733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:36:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:36:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781879514993456-2056-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781879514993456-2056-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:36:18.607290 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781879514993456-2056-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781879514993456-2056-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781879514993456-2056-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33287
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781879514993456-2056-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33287
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:36:18.719357 22746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:36:18.719705 22746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:36:18.719791 22746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:36:18.723949 22746 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:36:18.724072 22746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:36:18.724110 22746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:36:18.724148 22746 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:36:18.724177 22746 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:36:18.729599 22746 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781879514993456-2056-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781879514993456-2056-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33287
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781879514993456-2056-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781879514993456-2056-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:36:18.730769 22746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:36:18.731832 22746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:36:18.740660 22751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:18.740832 22754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:18.740839 22746 server_base.cc:1034] running on GCE node
W20260619 14:36:18.740717 22752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:18.741533 22746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:36:18.742760 22746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:36:18.743950 22746 hybrid_clock.cc:648] HybridClock initialized: now 1781879778743936 us; error 28 us; skew 500 ppm
Jun 19 14:36:18 dist-test-slave-70rm krb5kdc[22733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879778, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:36:18.746869 22746 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:36:18.747785 22746 webserver.cc:460] Webserver started at http://127.0.0.1:37561/ using document root <none> and password file <none>
I20260619 14:36:18.748366 22746 fs_manager.cc:359] Metadata directory not provided
I20260619 14:36:18.748461 22746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:36:18.748694 22746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:36:18.750852 22746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781879514993456-2056-0/minicluster-data/master-0/data/instance:
uuid: "6cc34f1081e74e96a49200ae5b7ae6b9"
format_stamp: "Formatted at 2026-06-19 14:36:18 on dist-test-slave-70rm"
server_key: "1728ad381ab3bf52ea859695678f8717"
server_key_iv: "4b872d2b0cdac81a021a1a240121ea25"
server_key_version: "encryptionkey@0"
I20260619 14:36:18.751415 22746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781879514993456-2056-0/minicluster-data/master-0/wal/instance:
uuid: "6cc34f1081e74e96a49200ae5b7ae6b9"
format_stamp: "Formatted at 2026-06-19 14:36:18 on dist-test-slave-70rm"
server_key: "1728ad381ab3bf52ea859695678f8717"
server_key_iv: "4b872d2b0cdac81a021a1a240121ea25"
server_key_version: "encryptionkey@0"
I20260619 14:36:18.755265 22746 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 14:36:18.757725 22761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:18.759001 22746 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:36:18.759151 22746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781879514993456-2056-0/minicluster-data/master-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781879514993456-2056-0/minicluster-data/master-0/wal
uuid: "6cc34f1081e74e96a49200ae5b7ae6b9"
format_stamp: "Formatted at 2026-06-19 14:36:18 on dist-test-slave-70rm"
server_key: "1728ad381ab3bf52ea859695678f8717"
server_key_iv: "4b872d2b0cdac81a021a1a240121ea25"
server_key_version: "encryptionkey@0"
I20260619 14:36:18.759290 22746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781879514993456-2056-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781879514993456-2056-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781879514993456-2056-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:36:18.783169 22746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:36:18.786609 22746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:36:18.786846 22746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:36:18.794063 22746 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33287
I20260619 14:36:18.794082 22813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33287 every 8 connection(s)
I20260619 14:36:18.795145 22746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
I20260619 14:36:18.798014 22814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:36:18.803615 22814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cc34f1081e74e96a49200ae5b7ae6b9: Bootstrap starting.
I20260619 14:36:18.804028 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 22746
I20260619 14:36:18.804204 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781879514993456-2056-0/minicluster-data/master-0/wal/instance
I20260619 14:36:18.804471 2056 external_mini_cluster.cc:1428] Setting key 3d02871230999578c0afbcbf4da5ad3d
I20260619 14:36:18.806288 22814 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6cc34f1081e74e96a49200ae5b7ae6b9: Neither blocks nor log segments found. Creating new log.
I20260619 14:36:18.806996 22814 log.cc:824] T 00000000000000000000000000000000 P 6cc34f1081e74e96a49200ae5b7ae6b9: Log is configured to *not* fsync() on all Append() calls
I20260619 14:36:18.808966 22814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cc34f1081e74e96a49200ae5b7ae6b9: No bootstrap required, opened a new log
I20260619 14:36:18.812640 22814 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6cc34f1081e74e96a49200ae5b7ae6b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cc34f1081e74e96a49200ae5b7ae6b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33287 } }
I20260619 14:36:18.812896 22814 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6cc34f1081e74e96a49200ae5b7ae6b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:36:18.812973 22814 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6cc34f1081e74e96a49200ae5b7ae6b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc34f1081e74e96a49200ae5b7ae6b9, State: Initialized, Role: FOLLOWER
I20260619 14:36:18.813433 22817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:18.805740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43795 (local address 127.0.0.1:33287)
0619 14:36:18.806178 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:18.806195 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:18.806231 (+ 36us) server_negotiation.cc:407] Connection header received
0619 14:36:18.806908 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:18.806925 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:18.807318 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:18.807657 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:18.808082 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:18.809193 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:18.810079 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:18.810608 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:18.810791 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:18.811670 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:18.811675 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:18.811687 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:36:18.812074 (+ 387us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:18.812126 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:36:18.812147 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:18.812262 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:18.812408 (+ 146us) server_negotiation.cc:299] Negotiation successful
0619 14:36:18.812610 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":136,"threads_started":1}
I20260619 14:36:18.813448 22814 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6cc34f1081e74e96a49200ae5b7ae6b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cc34f1081e74e96a49200ae5b7ae6b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33287 } }
I20260619 14:36:18.813701 22814 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6cc34f1081e74e96a49200ae5b7ae6b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:36:18.813776 22814 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6cc34f1081e74e96a49200ae5b7ae6b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:36:18.813863 22814 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6cc34f1081e74e96a49200ae5b7ae6b9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:36:18.814893 22814 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6cc34f1081e74e96a49200ae5b7ae6b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cc34f1081e74e96a49200ae5b7ae6b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33287 } }
I20260619 14:36:18.815263 22814 leader_election.cc:302] T 00000000000000000000000000000000 P 6cc34f1081e74e96a49200ae5b7ae6b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6cc34f1081e74e96a49200ae5b7ae6b9; no voters:
I20260619 14:36:18.815660 22814 leader_election.cc:288] T 00000000000000000000000000000000 P 6cc34f1081e74e96a49200ae5b7ae6b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:36:18.815697 22819 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6cc34f1081e74e96a49200ae5b7ae6b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:36:18.815930 22819 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6cc34f1081e74e96a49200ae5b7ae6b9 [term 1 LEADER]: Becoming Leader. State: Replica: 6cc34f1081e74e96a49200ae5b7ae6b9, State: Running, Role: LEADER
I20260619 14:36:18.816274 22819 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6cc34f1081e74e96a49200ae5b7ae6b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cc34f1081e74e96a49200ae5b7ae6b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33287 } }
I20260619 14:36:18.816329 2056 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:36:18.816941 22814 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6cc34f1081e74e96a49200ae5b7ae6b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:36:18.817405 22819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cc34f1081e74e96a49200ae5b7ae6b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6cc34f1081e74e96a49200ae5b7ae6b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cc34f1081e74e96a49200ae5b7ae6b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33287 } } }
I20260619 14:36:18.817570 22819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cc34f1081e74e96a49200ae5b7ae6b9 [sys.catalog]: This master's current role is: LEADER
I20260619 14:36:18.818001 22829 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:36:18.818562 22820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cc34f1081e74e96a49200ae5b7ae6b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cc34f1081e74e96a49200ae5b7ae6b9. Latest consensus state: current_term: 1 leader_uuid: "6cc34f1081e74e96a49200ae5b7ae6b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cc34f1081e74e96a49200ae5b7ae6b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33287 } } }
I20260619 14:36:18.818662 22820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cc34f1081e74e96a49200ae5b7ae6b9 [sys.catalog]: This master's current role is: LEADER
I20260619 14:36:18.821483 22829 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:36:18.824362 22817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:18.819756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53641 (local address 127.0.0.1:33287)
0619 14:36:18.819868 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:18.819872 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:18.820110 (+ 238us) server_negotiation.cc:407] Connection header received
0619 14:36:18.820311 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:18.820314 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:18.820363 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:18.820441 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:18.820730 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:18.821461 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:18.822562 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:18.822914 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:18.823027 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:18.823928 (+ 901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:18.823932 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:18.823935 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:36:18.823949 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:18.823978 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:36:18.823991 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:18.824070 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:18.824168 (+ 98us) server_negotiation.cc:299] Negotiation successful
0619 14:36:18.824220 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 14:36:18.827350 22829 catalog_manager.cc:1347] Generated new cluster ID: 7e92a2e74e0845bdac22a98ef06d6d85
I20260619 14:36:18.827418 22829 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:36:18.861265 22829 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:36:18.862643 22829 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:36:18.868049 22829 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6cc34f1081e74e96a49200ae5b7ae6b9: Generated new TSK 0
I20260619 14:36:18.868885 22829 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:36:18.873968 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 22746
2026-06-19T14:36:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1426 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781879514993456-2056-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:36:18 dist-test-slave-70rm krb5kdc[22845](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:36:18 dist-test-slave-70rm krb5kdc[22845](info): set up 2 sockets
Jun 19 14:36:18 dist-test-slave-70rm krb5kdc[22845](info): commencing operation
krb5kdc: starting...
W20260619 14:36:19.910288 2056 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Jun 19 14:36:19 dist-test-slave-70rm krb5kdc[22845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:36:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:36:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781879514993456-2056-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781879514993456-2056-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:36:20.032578 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781879514993456-2056-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781879514993456-2056-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781879514993456-2056-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45129
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781879514993456-2056-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45129
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:36:20.142220 22858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:36:20.142498 22858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:36:20.142540 22858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:36:20.146554 22858 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:36:20.146617 22858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:36:20.146636 22858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:36:20.146698 22858 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:36:20.146723 22858 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:36:20.152212 22858 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781879514993456-2056-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781879514993456-2056-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45129
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781879514993456-2056-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781879514993456-2056-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:36:20.153311 22858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:36:20.154377 22858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:36:20.163041 22863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:20.163048 22866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:20.163074 22864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:20.163419 22858 server_base.cc:1034] running on GCE node
I20260619 14:36:20.164044 22858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:36:20.165230 22858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:36:20.166440 22858 hybrid_clock.cc:648] HybridClock initialized: now 1781879780166431 us; error 36 us; skew 500 ppm
Jun 19 14:36:20 dist-test-slave-70rm krb5kdc[22845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879780, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:36:20.169346 22858 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:36:20.170315 22858 webserver.cc:460] Webserver started at http://127.0.0.1:41673/ using document root <none> and password file <none>
I20260619 14:36:20.170855 22858 fs_manager.cc:359] Metadata directory not provided
I20260619 14:36:20.170914 22858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:36:20.171082 22858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:36:20.173159 22858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781879514993456-2056-0/minicluster-data/master-0/data/instance:
uuid: "8a520edec85140b19f1a3fa873bddffb"
format_stamp: "Formatted at 2026-06-19 14:36:20 on dist-test-slave-70rm"
server_key: "c2b333110e3e854dc7501b46fe8b9ac6"
server_key_iv: "6d46d1860235a7297f43ffef788e5980"
server_key_version: "encryptionkey@0"
I20260619 14:36:20.173696 22858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781879514993456-2056-0/minicluster-data/master-0/wal/instance:
uuid: "8a520edec85140b19f1a3fa873bddffb"
format_stamp: "Formatted at 2026-06-19 14:36:20 on dist-test-slave-70rm"
server_key: "c2b333110e3e854dc7501b46fe8b9ac6"
server_key_iv: "6d46d1860235a7297f43ffef788e5980"
server_key_version: "encryptionkey@0"
I20260619 14:36:20.177620 22858 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260619 14:36:20.180192 22873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:20.181470 22858 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:36:20.181613 22858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781879514993456-2056-0/minicluster-data/master-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781879514993456-2056-0/minicluster-data/master-0/wal
uuid: "8a520edec85140b19f1a3fa873bddffb"
format_stamp: "Formatted at 2026-06-19 14:36:20 on dist-test-slave-70rm"
server_key: "c2b333110e3e854dc7501b46fe8b9ac6"
server_key_iv: "6d46d1860235a7297f43ffef788e5980"
server_key_version: "encryptionkey@0"
I20260619 14:36:20.181751 22858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781879514993456-2056-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781879514993456-2056-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781879514993456-2056-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:36:20.186684 22858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:36:20.189986 22858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:36:20.190232 22858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:36:20.196712 22858 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45129
I20260619 14:36:20.196789 22925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45129 every 8 connection(s)
I20260619 14:36:20.197676 22858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
I20260619 14:36:20.198647 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 22858
I20260619 14:36:20.198803 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781879514993456-2056-0/minicluster-data/master-0/wal/instance
I20260619 14:36:20.199090 2056 external_mini_cluster.cc:1428] Setting key e899193b2414af67ed7a316cd4a1b0ec
I20260619 14:36:20.200830 22926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:36:20.204779 22929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:20.200429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43925 (local address 127.0.0.1:45129)
0619 14:36:20.200857 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:20.200877 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:20.201047 (+ 170us) server_negotiation.cc:407] Connection header received
0619 14:36:20.201911 (+ 864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:20.201935 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:20.202340 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:20.202723 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:20.203361 (+ 638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:20.203366 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:20.203381 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:36:20.203391 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:20.203453 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:36:20.203480 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:20.203621 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:20.203704 (+ 83us) server_negotiation.cc:299] Negotiation successful
0619 14:36:20.203854 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":107,"threads_started":1}
I20260619 14:36:20.207551 2056 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:36:20.207844 22926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a520edec85140b19f1a3fa873bddffb: Bootstrap starting.
I20260619 14:36:20.210273 22926 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a520edec85140b19f1a3fa873bddffb: Neither blocks nor log segments found. Creating new log.
I20260619 14:36:20.211093 22926 log.cc:824] T 00000000000000000000000000000000 P 8a520edec85140b19f1a3fa873bddffb: Log is configured to *not* fsync() on all Append() calls
I20260619 14:36:20.211355 22929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:20.209575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54947 (local address 127.0.0.1:45129)
0619 14:36:20.209704 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:20.209708 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:20.209847 (+ 139us) server_negotiation.cc:407] Connection header received
0619 14:36:20.209935 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:20.209937 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:20.209972 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:20.210055 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:20.210826 (+ 771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:20.210829 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:20.210832 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:36:20.210842 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:20.210876 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:36:20.210889 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:20.210976 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:20.211041 (+ 65us) server_negotiation.cc:299] Negotiation successful
0619 14:36:20.211100 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 14:36:20.213083 22926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a520edec85140b19f1a3fa873bddffb: No bootstrap required, opened a new log
I20260619 14:36:20.216661 22926 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a520edec85140b19f1a3fa873bddffb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a520edec85140b19f1a3fa873bddffb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45129 } }
I20260619 14:36:20.216909 22926 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a520edec85140b19f1a3fa873bddffb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:36:20.216984 22926 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a520edec85140b19f1a3fa873bddffb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a520edec85140b19f1a3fa873bddffb, State: Initialized, Role: FOLLOWER
I20260619 14:36:20.217448 22926 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a520edec85140b19f1a3fa873bddffb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a520edec85140b19f1a3fa873bddffb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45129 } }
I20260619 14:36:20.217571 22926 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a520edec85140b19f1a3fa873bddffb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:36:20.217654 22926 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a520edec85140b19f1a3fa873bddffb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:36:20.217785 22926 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a520edec85140b19f1a3fa873bddffb [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:36:20.218789 22926 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a520edec85140b19f1a3fa873bddffb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a520edec85140b19f1a3fa873bddffb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45129 } }
I20260619 14:36:20.219115 22926 leader_election.cc:302] T 00000000000000000000000000000000 P 8a520edec85140b19f1a3fa873bddffb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a520edec85140b19f1a3fa873bddffb; no voters:
I20260619 14:36:20.219437 22926 leader_election.cc:288] T 00000000000000000000000000000000 P 8a520edec85140b19f1a3fa873bddffb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:36:20.219578 22936 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a520edec85140b19f1a3fa873bddffb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:36:20.219856 22936 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a520edec85140b19f1a3fa873bddffb [term 1 LEADER]: Becoming Leader. State: Replica: 8a520edec85140b19f1a3fa873bddffb, State: Running, Role: LEADER
I20260619 14:36:20.220263 22936 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a520edec85140b19f1a3fa873bddffb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a520edec85140b19f1a3fa873bddffb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45129 } }
I20260619 14:36:20.220693 22926 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a520edec85140b19f1a3fa873bddffb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:36:20.221406 22936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a520edec85140b19f1a3fa873bddffb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a520edec85140b19f1a3fa873bddffb. Latest consensus state: current_term: 1 leader_uuid: "8a520edec85140b19f1a3fa873bddffb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a520edec85140b19f1a3fa873bddffb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45129 } } }
I20260619 14:36:20.221586 22936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a520edec85140b19f1a3fa873bddffb [sys.catalog]: This master's current role is: LEADER
I20260619 14:36:20.221834 22938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a520edec85140b19f1a3fa873bddffb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a520edec85140b19f1a3fa873bddffb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a520edec85140b19f1a3fa873bddffb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45129 } } }
I20260619 14:36:20.221925 22938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a520edec85140b19f1a3fa873bddffb [sys.catalog]: This master's current role is: LEADER
I20260619 14:36:20.222014 22945 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:36:20.224838 22945 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:36:20.229765 22945 catalog_manager.cc:1347] Generated new cluster ID: 30c4c964d5584216a185d163eb876e5c
I20260619 14:36:20.229835 22945 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:36:20.245839 22945 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:36:20.246277 22945 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:36:20.250705 22945 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a520edec85140b19f1a3fa873bddffb: Generated new TSK 0
I20260619 14:36:20.251457 22945 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:36:20.256882 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 22858
2026-06-19T14:36:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1383 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781879514993456-2056-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:36:20 dist-test-slave-70rm krb5kdc[22957](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:36:20 dist-test-slave-70rm krb5kdc[22957](info): set up 2 sockets
Jun 19 14:36:20 dist-test-slave-70rm krb5kdc[22957](info): commencing operation
krb5kdc: starting...
W20260619 14:36:21.290413 2056 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Jun 19 14:36:21 dist-test-slave-70rm krb5kdc[22957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:36:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:36:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781879514993456-2056-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781879514993456-2056-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:36:21.409039 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781879514993456-2056-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781879514993456-2056-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781879514993456-2056-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32787
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781879514993456-2056-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32787
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:36:21.520392 22970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:36:21.520730 22970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:36:21.520805 22970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:36:21.524861 22970 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:36:21.524968 22970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:36:21.525007 22970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:36:21.525045 22970 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:36:21.525082 22970 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:36:21.530452 22970 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781879514993456-2056-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781879514993456-2056-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32787
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781879514993456-2056-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781879514993456-2056-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:36:21.531608 22970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:36:21.532687 22970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:36:21.542030 22970 server_base.cc:1034] running on GCE node
W20260619 14:36:21.541978 22976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:21.541980 22975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:21.542001 22978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:21.542831 22970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:36:21.543982 22970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:36:21.545146 22970 hybrid_clock.cc:648] HybridClock initialized: now 1781879781545135 us; error 28 us; skew 500 ppm
Jun 19 14:36:21 dist-test-slave-70rm krb5kdc[22957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879781, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:36:21.547897 22970 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:36:21.548862 22970 webserver.cc:460] Webserver started at http://127.0.0.1:40891/ using document root <none> and password file <none>
I20260619 14:36:21.549427 22970 fs_manager.cc:359] Metadata directory not provided
I20260619 14:36:21.549516 22970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:36:21.549758 22970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:36:21.551896 22970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781879514993456-2056-0/minicluster-data/master-0/data/instance:
uuid: "f3a0c0ac91f14fc5983349f4e8277b6a"
format_stamp: "Formatted at 2026-06-19 14:36:21 on dist-test-slave-70rm"
server_key: "5ce13cf264428490ecb2540e01746df8"
server_key_iv: "0f4310d3dd2b9ff11c22a1645516cf1a"
server_key_version: "encryptionkey@0"
I20260619 14:36:21.552489 22970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781879514993456-2056-0/minicluster-data/master-0/wal/instance:
uuid: "f3a0c0ac91f14fc5983349f4e8277b6a"
format_stamp: "Formatted at 2026-06-19 14:36:21 on dist-test-slave-70rm"
server_key: "5ce13cf264428490ecb2540e01746df8"
server_key_iv: "0f4310d3dd2b9ff11c22a1645516cf1a"
server_key_version: "encryptionkey@0"
I20260619 14:36:21.556397 22970 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:36:21.558997 22985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:21.560261 22970 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 14:36:21.560423 22970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781879514993456-2056-0/minicluster-data/master-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781879514993456-2056-0/minicluster-data/master-0/wal
uuid: "f3a0c0ac91f14fc5983349f4e8277b6a"
format_stamp: "Formatted at 2026-06-19 14:36:21 on dist-test-slave-70rm"
server_key: "5ce13cf264428490ecb2540e01746df8"
server_key_iv: "0f4310d3dd2b9ff11c22a1645516cf1a"
server_key_version: "encryptionkey@0"
I20260619 14:36:21.560552 22970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781879514993456-2056-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781879514993456-2056-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781879514993456-2056-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:36:21.576368 22970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:36:21.579411 22970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:36:21.579615 22970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:36:21.588594 22970 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32787
I20260619 14:36:21.588829 23037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32787 every 8 connection(s)
I20260619 14:36:21.589608 22970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
I20260619 14:36:21.592430 23038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:36:21.595782 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 22970
I20260619 14:36:21.595958 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781879514993456-2056-0/minicluster-data/master-0/wal/instance
I20260619 14:36:21.596222 2056 external_mini_cluster.cc:1428] Setting key 76cb16d84e68aebac6987e242b5e47d2
I20260619 14:36:21.598317 23038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3a0c0ac91f14fc5983349f4e8277b6a: Bootstrap starting.
I20260619 14:36:21.600488 23038 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3a0c0ac91f14fc5983349f4e8277b6a: Neither blocks nor log segments found. Creating new log.
I20260619 14:36:21.601217 23038 log.cc:824] T 00000000000000000000000000000000 P f3a0c0ac91f14fc5983349f4e8277b6a: Log is configured to *not* fsync() on all Append() calls
I20260619 14:36:21.603108 23038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3a0c0ac91f14fc5983349f4e8277b6a: No bootstrap required, opened a new log
Jun 19 14:36:21 dist-test-slave-70rm krb5kdc[22957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 14:36:21.607231 23038 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3a0c0ac91f14fc5983349f4e8277b6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3a0c0ac91f14fc5983349f4e8277b6a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32787 } }
I20260619 14:36:21.607453 23038 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3a0c0ac91f14fc5983349f4e8277b6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:36:21.607520 23038 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3a0c0ac91f14fc5983349f4e8277b6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3a0c0ac91f14fc5983349f4e8277b6a, State: Initialized, Role: FOLLOWER
I20260619 14:36:21.608003 23038 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3a0c0ac91f14fc5983349f4e8277b6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3a0c0ac91f14fc5983349f4e8277b6a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32787 } }
I20260619 14:36:21.608157 23038 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3a0c0ac91f14fc5983349f4e8277b6a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:36:21.608239 23038 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3a0c0ac91f14fc5983349f4e8277b6a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:36:21.608388 23038 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3a0c0ac91f14fc5983349f4e8277b6a [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:36:21.609637 23038 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3a0c0ac91f14fc5983349f4e8277b6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3a0c0ac91f14fc5983349f4e8277b6a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32787 } }
I20260619 14:36:21.610074 23038 leader_election.cc:302] T 00000000000000000000000000000000 P f3a0c0ac91f14fc5983349f4e8277b6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3a0c0ac91f14fc5983349f4e8277b6a; no voters:
I20260619 14:36:21.610433 23038 leader_election.cc:288] T 00000000000000000000000000000000 P f3a0c0ac91f14fc5983349f4e8277b6a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:36:21.610651 23043 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3a0c0ac91f14fc5983349f4e8277b6a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:36:21.610913 23043 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3a0c0ac91f14fc5983349f4e8277b6a [term 1 LEADER]: Becoming Leader. State: Replica: f3a0c0ac91f14fc5983349f4e8277b6a, State: Running, Role: LEADER
I20260619 14:36:21.611019 23041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:21.597642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:44269 (local address 127.0.0.1:32787)
0619 14:36:21.598074 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:21.598089 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:21.598170 (+ 81us) server_negotiation.cc:407] Connection header received
0619 14:36:21.598851 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:21.598869 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:21.599224 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:21.599563 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:21.600362 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:21.601459 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:21.602380 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:21.602852 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:21.603031 (+ 179us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:21.605718 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:21.605723 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:21.605737 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:36:21.605764 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:21.607427 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:21.608035 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:21.608047 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:21.608052 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:21.608194 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:21.608640 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:21.608647 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:21.608649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:21.609180 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:21.609425 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:21.609716 (+ 291us) server_negotiation.cc:299] Negotiation successful
0619 14:36:21.609961 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":135,"threads_started":1}
I20260619 14:36:21.611244 23043 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3a0c0ac91f14fc5983349f4e8277b6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3a0c0ac91f14fc5983349f4e8277b6a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32787 } }
I20260619 14:36:21.611675 23038 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3a0c0ac91f14fc5983349f4e8277b6a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:36:21.612479 23043 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3a0c0ac91f14fc5983349f4e8277b6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3a0c0ac91f14fc5983349f4e8277b6a. Latest consensus state: current_term: 1 leader_uuid: "f3a0c0ac91f14fc5983349f4e8277b6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3a0c0ac91f14fc5983349f4e8277b6a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32787 } } }
I20260619 14:36:21.612672 23043 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3a0c0ac91f14fc5983349f4e8277b6a [sys.catalog]: This master's current role is: LEADER
I20260619 14:36:21.612920 23044 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3a0c0ac91f14fc5983349f4e8277b6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3a0c0ac91f14fc5983349f4e8277b6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3a0c0ac91f14fc5983349f4e8277b6a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32787 } } }
I20260619 14:36:21.613126 23052 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:36:21.613060 23044 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3a0c0ac91f14fc5983349f4e8277b6a [sys.catalog]: This master's current role is: LEADER
I20260619 14:36:21.614598 2056 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:36:21.617031 23052 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:36:21.623245 23052 catalog_manager.cc:1347] Generated new cluster ID: cdbf72a07e414c468b78e65f51ad2e0e
I20260619 14:36:21.623318 23052 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:36:21.632302 23041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:21.616443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:57453 (local address 127.0.0.1:32787)
0619 14:36:21.616582 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:21.616586 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:21.616723 (+ 137us) server_negotiation.cc:407] Connection header received
0619 14:36:21.616874 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:21.616877 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:21.616923 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:21.617018 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:21.617732 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:21.618374 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:21.619309 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:21.619548 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:21.619665 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:21.621275 (+ 1610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:21.621279 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:21.621281 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:36:21.621308 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:21.623073 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:21.630823 (+ 7750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:21.630828 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:21.630830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:21.630949 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:21.631332 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:21.631335 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:21.631337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:21.631563 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:21.631719 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:21.632017 (+ 298us) server_negotiation.cc:299] Negotiation successful
0619 14:36:21.632130 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 14:36:21.634465 23052 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:36:21.635349 23052 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:36:21.645769 23052 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3a0c0ac91f14fc5983349f4e8277b6a: Generated new TSK 0
I20260619 14:36:21.646574 23052 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:36:21.651283 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 22970
W20260619 14:36:21.656601 22955 connection.cc:535] client connection to 127.0.0.1:32787 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32787): Connection reset by peer (error 104)
2026-06-19T14:36:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1394 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781879514993456-2056-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:36:21 dist-test-slave-70rm krb5kdc[23069](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:36:21 dist-test-slave-70rm krb5kdc[23069](info): set up 2 sockets
Jun 19 14:36:21 dist-test-slave-70rm krb5kdc[23069](info): commencing operation
krb5kdc: starting...
W20260619 14:36:22.702373 2056 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Jun 19 14:36:22 dist-test-slave-70rm krb5kdc[23069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:36:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:36:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781879514993456-2056-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781879514993456-2056-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:36:22.822678 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781879514993456-2056-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781879514993456-2056-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781879514993456-2056-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39817
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781879514993456-2056-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39817
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:36:22.937986 23082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:36:22.938361 23082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:36:22.938434 23082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:36:22.942548 23082 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:36:22.942611 23082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:36:22.942631 23082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:36:22.942698 23082 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:36:22.942725 23082 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:36:22.948158 23082 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781879514993456-2056-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781879514993456-2056-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781879514993456-2056-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781879514993456-2056-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:36:22.949234 23082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:36:22.950289 23082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:36:22.958635 23090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:22.958590 23087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:22.958705 23082 server_base.cc:1034] running on GCE node
W20260619 14:36:22.958590 23088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:22.959417 23082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:36:22.960532 23082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:36:22.961728 23082 hybrid_clock.cc:648] HybridClock initialized: now 1781879782961693 us; error 50 us; skew 500 ppm
Jun 19 14:36:22 dist-test-slave-70rm krb5kdc[23069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879782, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:36:22.964533 23082 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:36:22.965456 23082 webserver.cc:460] Webserver started at http://127.0.0.1:36029/ using document root <none> and password file <none>
I20260619 14:36:22.966008 23082 fs_manager.cc:359] Metadata directory not provided
I20260619 14:36:22.966099 23082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:36:22.966398 23082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:36:22.968428 23082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781879514993456-2056-0/minicluster-data/master-0/data/instance:
uuid: "7bfe47bd4a224feb964df211063cd851"
format_stamp: "Formatted at 2026-06-19 14:36:22 on dist-test-slave-70rm"
server_key: "734357fee24947fda388ac980589b68e"
server_key_iv: "6bd0c4baf0aa971fe669704b3f7374c2"
server_key_version: "encryptionkey@0"
I20260619 14:36:22.968994 23082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781879514993456-2056-0/minicluster-data/master-0/wal/instance:
uuid: "7bfe47bd4a224feb964df211063cd851"
format_stamp: "Formatted at 2026-06-19 14:36:22 on dist-test-slave-70rm"
server_key: "734357fee24947fda388ac980589b68e"
server_key_iv: "6bd0c4baf0aa971fe669704b3f7374c2"
server_key_version: "encryptionkey@0"
I20260619 14:36:22.972741 23082 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 14:36:22.975298 23097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:22.976531 23082 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:36:22.976677 23082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781879514993456-2056-0/minicluster-data/master-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781879514993456-2056-0/minicluster-data/master-0/wal
uuid: "7bfe47bd4a224feb964df211063cd851"
format_stamp: "Formatted at 2026-06-19 14:36:22 on dist-test-slave-70rm"
server_key: "734357fee24947fda388ac980589b68e"
server_key_iv: "6bd0c4baf0aa971fe669704b3f7374c2"
server_key_version: "encryptionkey@0"
I20260619 14:36:22.976799 23082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781879514993456-2056-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781879514993456-2056-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781879514993456-2056-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:36:23.021121 23082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:36:23.024578 23082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:36:23.024791 23082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:36:23.032014 23082 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39817
I20260619 14:36:23.032043 23149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39817 every 8 connection(s)
I20260619 14:36:23.032961 23082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
I20260619 14:36:23.035823 23150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:36:23.039666 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 23082
I20260619 14:36:23.039824 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781879514993456-2056-0/minicluster-data/master-0/wal/instance
I20260619 14:36:23.040084 2056 external_mini_cluster.cc:1428] Setting key 59697dd4c8636dd789a286b22fa39ca4
I20260619 14:36:23.042088 23150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bfe47bd4a224feb964df211063cd851: Bootstrap starting.
I20260619 14:36:23.044425 23150 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7bfe47bd4a224feb964df211063cd851: Neither blocks nor log segments found. Creating new log.
I20260619 14:36:23.045197 23150 log.cc:824] T 00000000000000000000000000000000 P 7bfe47bd4a224feb964df211063cd851: Log is configured to *not* fsync() on all Append() calls
I20260619 14:36:23.047240 23150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bfe47bd4a224feb964df211063cd851: No bootstrap required, opened a new log
I20260619 14:36:23.049474 23153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:23.041565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:45471 (local address 127.0.0.1:39817)
0619 14:36:23.042006 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:23.042022 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:23.042057 (+ 35us) server_negotiation.cc:407] Connection header received
0619 14:36:23.042813 (+ 756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:23.042830 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:23.043188 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:23.043546 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:23.043968 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:23.045015 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:23.045965 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:23.046520 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:23.046685 (+ 165us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:23.047591 (+ 906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:23.047596 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:23.047610 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:36:23.047993 (+ 383us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:23.048046 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:36:23.048068 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:23.048213 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:23.048356 (+ 143us) server_negotiation.cc:299] Negotiation successful
0619 14:36:23.048496 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":132,"threads_started":1}
I20260619 14:36:23.050480 23150 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7bfe47bd4a224feb964df211063cd851 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bfe47bd4a224feb964df211063cd851" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39817 } }
I20260619 14:36:23.050770 23150 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7bfe47bd4a224feb964df211063cd851 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:36:23.050833 23150 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7bfe47bd4a224feb964df211063cd851 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bfe47bd4a224feb964df211063cd851, State: Initialized, Role: FOLLOWER
I20260619 14:36:23.051276 23150 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7bfe47bd4a224feb964df211063cd851 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bfe47bd4a224feb964df211063cd851" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39817 } }
I20260619 14:36:23.051451 23150 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7bfe47bd4a224feb964df211063cd851 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:36:23.051522 23150 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7bfe47bd4a224feb964df211063cd851 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:36:23.051617 23150 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7bfe47bd4a224feb964df211063cd851 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:36:23.052472 2056 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:36:23.052812 23150 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7bfe47bd4a224feb964df211063cd851 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bfe47bd4a224feb964df211063cd851" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39817 } }
I20260619 14:36:23.053174 23150 leader_election.cc:302] T 00000000000000000000000000000000 P 7bfe47bd4a224feb964df211063cd851 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7bfe47bd4a224feb964df211063cd851; no voters:
I20260619 14:36:23.053499 23150 leader_election.cc:288] T 00000000000000000000000000000000 P 7bfe47bd4a224feb964df211063cd851 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:36:23.053576 23155 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7bfe47bd4a224feb964df211063cd851 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:36:23.053814 23155 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7bfe47bd4a224feb964df211063cd851 [term 1 LEADER]: Becoming Leader. State: Replica: 7bfe47bd4a224feb964df211063cd851, State: Running, Role: LEADER
I20260619 14:36:23.054170 23155 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7bfe47bd4a224feb964df211063cd851 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bfe47bd4a224feb964df211063cd851" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39817 } }
I20260619 14:36:23.054802 23150 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7bfe47bd4a224feb964df211063cd851 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:36:23.055872 23155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bfe47bd4a224feb964df211063cd851 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7bfe47bd4a224feb964df211063cd851. Latest consensus state: current_term: 1 leader_uuid: "7bfe47bd4a224feb964df211063cd851" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bfe47bd4a224feb964df211063cd851" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39817 } } }
I20260619 14:36:23.055984 23155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bfe47bd4a224feb964df211063cd851 [sys.catalog]: This master's current role is: LEADER
I20260619 14:36:23.056018 23158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bfe47bd4a224feb964df211063cd851 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7bfe47bd4a224feb964df211063cd851" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bfe47bd4a224feb964df211063cd851" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39817 } } }
I20260619 14:36:23.056133 23158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bfe47bd4a224feb964df211063cd851 [sys.catalog]: This master's current role is: LEADER
I20260619 14:36:23.057298 23169 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:36:23.059943 23153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:23.054606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:41651 (local address 127.0.0.1:39817)
0619 14:36:23.054727 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:23.054732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:23.055122 (+ 390us) server_negotiation.cc:407] Connection header received
0619 14:36:23.055297 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:23.055300 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:23.055348 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:23.055414 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:23.055783 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:23.056983 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:23.058111 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:23.058459 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:23.058565 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:23.059445 (+ 880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:23.059449 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:23.059451 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:36:23.059464 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:23.059499 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:36:23.059510 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:23.059609 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:23.059763 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 14:36:23.059799 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 14:36:23.060774 23169 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:36:23.066416 23169 catalog_manager.cc:1347] Generated new cluster ID: ef1aff3eac5742408cfe3de87c93ec7a
I20260619 14:36:23.066504 23169 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:36:23.076372 23169 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:36:23.077301 23169 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:36:23.087492 23169 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7bfe47bd4a224feb964df211063cd851: Generated new TSK 0
I20260619 14:36:23.088459 23169 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:36:23.097990 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 23082
W20260619 14:36:23.103225 23067 connection.cc:535] client connection to 127.0.0.1:39817 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39817): Connection reset by peer (error 104)
2026-06-19T14:36:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1446 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781879514993456-2056-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:36:23 dist-test-slave-70rm krb5kdc[23181](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:36:23 dist-test-slave-70rm krb5kdc[23181](info): set up 2 sockets
Jun 19 14:36:23 dist-test-slave-70rm krb5kdc[23181](info): commencing operation
krb5kdc: starting...
W20260619 14:36:24.170547 2056 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Jun 19 14:36:24 dist-test-slave-70rm krb5kdc[23181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:36:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:36:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781879514993456-2056-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781879514993456-2056-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:36:24.291035 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781879514993456-2056-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781879514993456-2056-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781879514993456-2056-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39399
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781879514993456-2056-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39399
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:36:24.401484 23194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:36:24.401760 23194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:36:24.401803 23194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:36:24.405857 23194 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:36:24.405922 23194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:36:24.405942 23194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:36:24.405961 23194 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:36:24.405975 23194 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:36:24.411422 23194 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781879514993456-2056-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781879514993456-2056-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39399
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781879514993456-2056-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781879514993456-2056-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:36:24.412513 23194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:36:24.413578 23194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:36:24.422201 23200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:24.422211 23199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:24.422204 23202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:24.422379 23194 server_base.cc:1034] running on GCE node
I20260619 14:36:24.423060 23194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:36:24.424185 23194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:36:24.425381 23194 hybrid_clock.cc:648] HybridClock initialized: now 1781879784425357 us; error 45 us; skew 500 ppm
Jun 19 14:36:24 dist-test-slave-70rm krb5kdc[23181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:36:24.428229 23194 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:36:24.429215 23194 webserver.cc:460] Webserver started at http://127.0.0.1:36875/ using document root <none> and password file <none>
I20260619 14:36:24.429899 23194 fs_manager.cc:359] Metadata directory not provided
I20260619 14:36:24.430047 23194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:36:24.430357 23194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:36:24.433135 23194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781879514993456-2056-0/minicluster-data/master-0/data/instance:
uuid: "7e47637f9a5e471eac7bb3f73c261285"
format_stamp: "Formatted at 2026-06-19 14:36:24 on dist-test-slave-70rm"
server_key: "8ad152de7f7b66054c7b4d3ce49dc135"
server_key_iv: "6df2ec5f31d0f1b23f9bc6b60591a6c9"
server_key_version: "encryptionkey@0"
I20260619 14:36:24.433723 23194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781879514993456-2056-0/minicluster-data/master-0/wal/instance:
uuid: "7e47637f9a5e471eac7bb3f73c261285"
format_stamp: "Formatted at 2026-06-19 14:36:24 on dist-test-slave-70rm"
server_key: "8ad152de7f7b66054c7b4d3ce49dc135"
server_key_iv: "6df2ec5f31d0f1b23f9bc6b60591a6c9"
server_key_version: "encryptionkey@0"
I20260619 14:36:24.437561 23194 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 14:36:24.440053 23209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:24.441260 23194 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:36:24.441398 23194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781879514993456-2056-0/minicluster-data/master-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781879514993456-2056-0/minicluster-data/master-0/wal
uuid: "7e47637f9a5e471eac7bb3f73c261285"
format_stamp: "Formatted at 2026-06-19 14:36:24 on dist-test-slave-70rm"
server_key: "8ad152de7f7b66054c7b4d3ce49dc135"
server_key_iv: "6df2ec5f31d0f1b23f9bc6b60591a6c9"
server_key_version: "encryptionkey@0"
I20260619 14:36:24.441510 23194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781879514993456-2056-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781879514993456-2056-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781879514993456-2056-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:36:24.446045 23194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:36:24.448822 23194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:36:24.449025 23194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:36:24.456082 23194 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39399
I20260619 14:36:24.456066 23261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39399 every 8 connection(s)
I20260619 14:36:24.457019 23194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
I20260619 14:36:24.459700 23262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:36:24.465080 23262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e47637f9a5e471eac7bb3f73c261285: Bootstrap starting.
I20260619 14:36:24.467074 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 23194
I20260619 14:36:24.467211 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781879514993456-2056-0/minicluster-data/master-0/wal/instance
I20260619 14:36:24.467496 2056 external_mini_cluster.cc:1428] Setting key a0fb78f455514c2f66516716ceb7eb1f
I20260619 14:36:24.467748 23262 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e47637f9a5e471eac7bb3f73c261285: Neither blocks nor log segments found. Creating new log.
I20260619 14:36:24.468554 23262 log.cc:824] T 00000000000000000000000000000000 P 7e47637f9a5e471eac7bb3f73c261285: Log is configured to *not* fsync() on all Append() calls
I20260619 14:36:24.470628 23262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e47637f9a5e471eac7bb3f73c261285: No bootstrap required, opened a new log
I20260619 14:36:24.472498 23265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:24.468652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:42151 (local address 127.0.0.1:39399)
0619 14:36:24.469148 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:24.469167 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:24.469211 (+ 44us) server_negotiation.cc:407] Connection header received
0619 14:36:24.469928 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:24.469947 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:24.470293 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:24.470621 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:24.471367 (+ 746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:24.471372 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:24.471383 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:36:24.471393 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:24.471443 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:36:24.471461 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:24.471573 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:24.471671 (+ 98us) server_negotiation.cc:299] Negotiation successful
0619 14:36:24.471839 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":144,"threads_started":1}
I20260619 14:36:24.474695 23262 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e47637f9a5e471eac7bb3f73c261285 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e47637f9a5e471eac7bb3f73c261285" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39399 } }
I20260619 14:36:24.474992 23262 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e47637f9a5e471eac7bb3f73c261285 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:36:24.475051 23262 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e47637f9a5e471eac7bb3f73c261285 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e47637f9a5e471eac7bb3f73c261285, State: Initialized, Role: FOLLOWER
I20260619 14:36:24.475340 2056 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:36:24.475503 23262 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e47637f9a5e471eac7bb3f73c261285 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e47637f9a5e471eac7bb3f73c261285" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39399 } }
I20260619 14:36:24.475654 23262 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e47637f9a5e471eac7bb3f73c261285 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:36:24.475747 23262 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e47637f9a5e471eac7bb3f73c261285 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:36:24.475845 23262 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e47637f9a5e471eac7bb3f73c261285 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:36:24.476850 23262 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e47637f9a5e471eac7bb3f73c261285 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e47637f9a5e471eac7bb3f73c261285" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39399 } }
I20260619 14:36:24.477227 23262 leader_election.cc:302] T 00000000000000000000000000000000 P 7e47637f9a5e471eac7bb3f73c261285 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e47637f9a5e471eac7bb3f73c261285; no voters:
I20260619 14:36:24.477587 23262 leader_election.cc:288] T 00000000000000000000000000000000 P 7e47637f9a5e471eac7bb3f73c261285 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:36:24.477732 23271 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e47637f9a5e471eac7bb3f73c261285 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:36:24.477952 23271 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e47637f9a5e471eac7bb3f73c261285 [term 1 LEADER]: Becoming Leader. State: Replica: 7e47637f9a5e471eac7bb3f73c261285, State: Running, Role: LEADER
I20260619 14:36:24.478366 23271 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e47637f9a5e471eac7bb3f73c261285 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e47637f9a5e471eac7bb3f73c261285" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39399 } }
I20260619 14:36:24.479020 23262 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e47637f9a5e471eac7bb3f73c261285 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:36:24.479581 23265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:24.477513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:58541 (local address 127.0.0.1:39399)
0619 14:36:24.477618 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:24.477622 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:24.477833 (+ 211us) server_negotiation.cc:407] Connection header received
0619 14:36:24.478035 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:24.478041 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:24.478077 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:24.478158 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:24.478984 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:24.478988 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:24.478990 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:36:24.479002 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:24.479034 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:36:24.479051 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:24.479138 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:24.479425 (+ 287us) server_negotiation.cc:299] Negotiation successful
0619 14:36:24.479466 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 14:36:24.480100 23273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e47637f9a5e471eac7bb3f73c261285 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e47637f9a5e471eac7bb3f73c261285" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e47637f9a5e471eac7bb3f73c261285" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39399 } } }
I20260619 14:36:24.480234 23273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e47637f9a5e471eac7bb3f73c261285 [sys.catalog]: This master's current role is: LEADER
I20260619 14:36:24.480549 23274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e47637f9a5e471eac7bb3f73c261285 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e47637f9a5e471eac7bb3f73c261285. Latest consensus state: current_term: 1 leader_uuid: "7e47637f9a5e471eac7bb3f73c261285" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e47637f9a5e471eac7bb3f73c261285" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39399 } } }
I20260619 14:36:24.480669 23274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e47637f9a5e471eac7bb3f73c261285 [sys.catalog]: This master's current role is: LEADER
W20260619 14:36:24.483723 23287 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7e47637f9a5e471eac7bb3f73c261285: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:36:24.483791 23287 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:36:24.483873 23281 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:36:24.484536 23281 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:36:24.490101 23281 catalog_manager.cc:1347] Generated new cluster ID: 9fc659e71de94accb73b16ca4ea416aa
I20260619 14:36:24.490278 23281 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:36:24.500267 23281 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:36:24.500568 23281 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:36:24.508139 23281 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e47637f9a5e471eac7bb3f73c261285: Generated new TSK 0
I20260619 14:36:24.508894 23281 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:36:24.516383 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 23194
2026-06-19T14:36:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1418 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781879514993456-2056-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:36:24 dist-test-slave-70rm krb5kdc[23293](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:36:24 dist-test-slave-70rm krb5kdc[23293](info): set up 2 sockets
Jun 19 14:36:24 dist-test-slave-70rm krb5kdc[23293](info): commencing operation
krb5kdc: starting...
W20260619 14:36:25.542498 2056 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Jun 19 14:36:25 dist-test-slave-70rm krb5kdc[23293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:36:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:36:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781879514993456-2056-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781879514993456-2056-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:36:25.667472 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781879514993456-2056-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781879514993456-2056-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781879514993456-2056-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38631
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781879514993456-2056-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38631
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:36:25.780331 23306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:36:25.780656 23306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:36:25.780733 23306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:36:25.784958 23306 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:36:25.785049 23306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:36:25.785087 23306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:36:25.785123 23306 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:36:25.785162 23306 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:36:25.790665 23306 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781879514993456-2056-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781879514993456-2056-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38631
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781879514993456-2056-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781879514993456-2056-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:36:25.791800 23306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:36:25.792862 23306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:36:25.801390 23312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:25.801406 23314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:25.801534 23306 server_base.cc:1034] running on GCE node
W20260619 14:36:25.801724 23311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:25.802341 23306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:36:25.803488 23306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:36:25.804682 23306 hybrid_clock.cc:648] HybridClock initialized: now 1781879785804669 us; error 31 us; skew 500 ppm
Jun 19 14:36:25 dist-test-slave-70rm krb5kdc[23293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879785, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:36:25.807507 23306 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:36:25.808434 23306 webserver.cc:460] Webserver started at http://127.0.0.1:42489/ using document root <none> and password file <none>
I20260619 14:36:25.809008 23306 fs_manager.cc:359] Metadata directory not provided
I20260619 14:36:25.809100 23306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:36:25.809332 23306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:36:25.811414 23306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781879514993456-2056-0/minicluster-data/master-0/data/instance:
uuid: "38d7d2312e664f64a60fb4023df44ab0"
format_stamp: "Formatted at 2026-06-19 14:36:25 on dist-test-slave-70rm"
server_key: "b10b240c60fd98fe48a59c5ea142bb68"
server_key_iv: "c0e35ff4ca34d48325fdfa3cb8b69431"
server_key_version: "encryptionkey@0"
I20260619 14:36:25.811959 23306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781879514993456-2056-0/minicluster-data/master-0/wal/instance:
uuid: "38d7d2312e664f64a60fb4023df44ab0"
format_stamp: "Formatted at 2026-06-19 14:36:25 on dist-test-slave-70rm"
server_key: "b10b240c60fd98fe48a59c5ea142bb68"
server_key_iv: "c0e35ff4ca34d48325fdfa3cb8b69431"
server_key_version: "encryptionkey@0"
I20260619 14:36:25.815555 23306 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:36:25.817847 23321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:25.819072 23306 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:36:25.819218 23306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781879514993456-2056-0/minicluster-data/master-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781879514993456-2056-0/minicluster-data/master-0/wal
uuid: "38d7d2312e664f64a60fb4023df44ab0"
format_stamp: "Formatted at 2026-06-19 14:36:25 on dist-test-slave-70rm"
server_key: "b10b240c60fd98fe48a59c5ea142bb68"
server_key_iv: "c0e35ff4ca34d48325fdfa3cb8b69431"
server_key_version: "encryptionkey@0"
I20260619 14:36:25.819334 23306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781879514993456-2056-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781879514993456-2056-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781879514993456-2056-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:36:25.835304 23306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:36:25.838815 23306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:36:25.839035 23306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:36:25.846276 23306 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38631
I20260619 14:36:25.846292 23373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38631 every 8 connection(s)
I20260619 14:36:25.847240 23306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
I20260619 14:36:25.849877 23374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:36:25.854166 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 23306
I20260619 14:36:25.854313 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781879514993456-2056-0/minicluster-data/master-0/wal/instance
I20260619 14:36:25.854557 2056 external_mini_cluster.cc:1428] Setting key 9b210e264ad7b2d4628fb6748b689142
I20260619 14:36:25.855579 23374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38d7d2312e664f64a60fb4023df44ab0: Bootstrap starting.
I20260619 14:36:25.857864 23374 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38d7d2312e664f64a60fb4023df44ab0: Neither blocks nor log segments found. Creating new log.
I20260619 14:36:25.858678 23374 log.cc:824] T 00000000000000000000000000000000 P 38d7d2312e664f64a60fb4023df44ab0: Log is configured to *not* fsync() on all Append() calls
I20260619 14:36:25.860543 23374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38d7d2312e664f64a60fb4023df44ab0: No bootstrap required, opened a new log
Jun 19 14:36:25 dist-test-slave-70rm krb5kdc[23293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 14:36:25.864637 23374 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38d7d2312e664f64a60fb4023df44ab0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38d7d2312e664f64a60fb4023df44ab0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38631 } }
I20260619 14:36:25.864861 23374 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38d7d2312e664f64a60fb4023df44ab0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:36:25.864923 23374 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38d7d2312e664f64a60fb4023df44ab0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38d7d2312e664f64a60fb4023df44ab0, State: Initialized, Role: FOLLOWER
I20260619 14:36:25.865468 23374 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38d7d2312e664f64a60fb4023df44ab0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38d7d2312e664f64a60fb4023df44ab0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38631 } }
I20260619 14:36:25.865633 23374 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38d7d2312e664f64a60fb4023df44ab0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:36:25.865705 23374 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38d7d2312e664f64a60fb4023df44ab0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:36:25.865801 23374 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38d7d2312e664f64a60fb4023df44ab0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:36:25.866997 23374 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38d7d2312e664f64a60fb4023df44ab0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38d7d2312e664f64a60fb4023df44ab0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38631 } }
I20260619 14:36:25.867376 23374 leader_election.cc:302] T 00000000000000000000000000000000 P 38d7d2312e664f64a60fb4023df44ab0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38d7d2312e664f64a60fb4023df44ab0; no voters:
I20260619 14:36:25.867733 23374 leader_election.cc:288] T 00000000000000000000000000000000 P 38d7d2312e664f64a60fb4023df44ab0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:36:25.867859 23379 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38d7d2312e664f64a60fb4023df44ab0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:36:25.868113 23379 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38d7d2312e664f64a60fb4023df44ab0 [term 1 LEADER]: Becoming Leader. State: Replica: 38d7d2312e664f64a60fb4023df44ab0, State: Running, Role: LEADER
I20260619 14:36:25.868502 23379 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38d7d2312e664f64a60fb4023df44ab0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38d7d2312e664f64a60fb4023df44ab0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38631 } }
I20260619 14:36:25.869029 23374 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38d7d2312e664f64a60fb4023df44ab0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:36:25.869807 23377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:25.855983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:33827 (local address 127.0.0.1:38631)
0619 14:36:25.856389 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:25.856403 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:25.856438 (+ 35us) server_negotiation.cc:407] Connection header received
0619 14:36:25.857050 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:25.857066 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:25.857416 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:25.857759 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:25.858587 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:25.859623 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:25.860494 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:25.861015 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:25.861181 (+ 166us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:25.864134 (+ 2953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:25.864139 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:25.864154 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:36:25.864181 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:25.866381 (+ 2200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:25.866917 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:25.866926 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:25.866935 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:25.867090 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:25.867430 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:25.867433 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:25.867435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:25.867974 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:25.868188 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:25.868558 (+ 370us) server_negotiation.cc:299] Negotiation successful
0619 14:36:25.868816 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":130,"threads_started":1}
I20260619 14:36:25.870178 23380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38d7d2312e664f64a60fb4023df44ab0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38d7d2312e664f64a60fb4023df44ab0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38d7d2312e664f64a60fb4023df44ab0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38631 } } }
I20260619 14:36:25.870307 23380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38d7d2312e664f64a60fb4023df44ab0 [sys.catalog]: This master's current role is: LEADER
I20260619 14:36:25.871542 23381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38d7d2312e664f64a60fb4023df44ab0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38d7d2312e664f64a60fb4023df44ab0. Latest consensus state: current_term: 1 leader_uuid: "38d7d2312e664f64a60fb4023df44ab0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38d7d2312e664f64a60fb4023df44ab0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38631 } } }
I20260619 14:36:25.871693 23381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38d7d2312e664f64a60fb4023df44ab0 [sys.catalog]: This master's current role is: LEADER
I20260619 14:36:25.873654 2056 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260619 14:36:25.873970 23394 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 38d7d2312e664f64a60fb4023df44ab0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:36:25.874055 23394 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:36:25.874191 23388 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:36:25.874888 23388 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:36:25.881290 23388 catalog_manager.cc:1347] Generated new cluster ID: afd9cc3422b9444ea4e0db1875431812
I20260619 14:36:25.881402 23388 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:36:25.885083 23377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:25.875960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:43017 (local address 127.0.0.1:38631)
0619 14:36:25.876080 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:25.876085 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:25.876282 (+ 197us) server_negotiation.cc:407] Connection header received
0619 14:36:25.876410 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:25.876413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:25.876467 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:25.876553 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:25.877415 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:25.878203 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:25.879341 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:25.879656 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:25.879763 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:25.881550 (+ 1787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:25.881553 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:25.881556 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:36:25.881580 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:25.883270 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:25.883788 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:25.883791 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:25.883793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:25.883868 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:25.884179 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:25.884182 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:25.884183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:25.884387 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:25.884484 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:25.884842 (+ 358us) server_negotiation.cc:299] Negotiation successful
0619 14:36:25.884932 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 14:36:25.892829 23388 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:36:25.893782 23388 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:36:25.901887 23388 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38d7d2312e664f64a60fb4023df44ab0: Generated new TSK 0
I20260619 14:36:25.902647 23388 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:36:25.908149 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 23306
W20260619 14:36:25.913429 23291 connection.cc:535] client connection to 127.0.0.1:38631 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38631): Connection reset by peer (error 104)
2026-06-19T14:36:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1391 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781879514993456-2056-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:36:25 dist-test-slave-70rm krb5kdc[23405](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:36:25 dist-test-slave-70rm krb5kdc[23405](info): set up 2 sockets
Jun 19 14:36:25 dist-test-slave-70rm krb5kdc[23405](info): commencing operation
krb5kdc: starting...
W20260619 14:36:26.938349 2056 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Jun 19 14:36:27 dist-test-slave-70rm krb5kdc[23405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:36:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:36:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781879514993456-2056-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781879514993456-2056-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:36:27.059537 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781879514993456-2056-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781879514993456-2056-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781879514993456-2056-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32979
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781879514993456-2056-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32979
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:36:27.170414 23418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:36:27.170706 23418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:36:27.170783 23418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:36:27.174989 23418 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:36:27.175055 23418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:36:27.175074 23418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:36:27.175129 23418 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:36:27.175163 23418 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:36:27.180531 23418 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781879514993456-2056-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781879514993456-2056-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32979
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781879514993456-2056-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781879514993456-2056-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:36:27.181640 23418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:36:27.182744 23418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:36:27.191493 23423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:27.191557 23424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:27.191493 23426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:27.191784 23418 server_base.cc:1034] running on GCE node
I20260619 14:36:27.192463 23418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:36:27.193650 23418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:36:27.194842 23418 hybrid_clock.cc:648] HybridClock initialized: now 1781879787194832 us; error 41 us; skew 500 ppm
Jun 19 14:36:27 dist-test-slave-70rm krb5kdc[23405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879787, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:36:27.197553 23418 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:36:27.198577 23418 webserver.cc:460] Webserver started at http://127.0.0.1:34691/ using document root <none> and password file <none>
I20260619 14:36:27.199147 23418 fs_manager.cc:359] Metadata directory not provided
I20260619 14:36:27.199240 23418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:36:27.199488 23418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:36:27.201593 23418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781879514993456-2056-0/minicluster-data/master-0/data/instance:
uuid: "e7d2129de5c347b587828bc4e5fe29e8"
format_stamp: "Formatted at 2026-06-19 14:36:27 on dist-test-slave-70rm"
server_key: "c78b98de69faeb19e1a094343b9c0f71"
server_key_iv: "12758c83626dcc351fab7e365a981b6e"
server_key_version: "encryptionkey@0"
I20260619 14:36:27.202236 23418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781879514993456-2056-0/minicluster-data/master-0/wal/instance:
uuid: "e7d2129de5c347b587828bc4e5fe29e8"
format_stamp: "Formatted at 2026-06-19 14:36:27 on dist-test-slave-70rm"
server_key: "c78b98de69faeb19e1a094343b9c0f71"
server_key_iv: "12758c83626dcc351fab7e365a981b6e"
server_key_version: "encryptionkey@0"
I20260619 14:36:27.205981 23418 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:36:27.208427 23433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:27.209652 23418 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:36:27.209815 23418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781879514993456-2056-0/minicluster-data/master-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781879514993456-2056-0/minicluster-data/master-0/wal
uuid: "e7d2129de5c347b587828bc4e5fe29e8"
format_stamp: "Formatted at 2026-06-19 14:36:27 on dist-test-slave-70rm"
server_key: "c78b98de69faeb19e1a094343b9c0f71"
server_key_iv: "12758c83626dcc351fab7e365a981b6e"
server_key_version: "encryptionkey@0"
I20260619 14:36:27.209954 23418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781879514993456-2056-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781879514993456-2056-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781879514993456-2056-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:36:27.224110 23418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:36:27.227329 23418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:36:27.227624 23418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:36:27.234869 23418 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32979
I20260619 14:36:27.234936 23485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32979 every 8 connection(s)
I20260619 14:36:27.235971 23418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
I20260619 14:36:27.238885 23486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:36:27.244395 23486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7d2129de5c347b587828bc4e5fe29e8: Bootstrap starting.
I20260619 14:36:27.246047 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 23418
I20260619 14:36:27.246233 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781879514993456-2056-0/minicluster-data/master-0/wal/instance
I20260619 14:36:27.246493 2056 external_mini_cluster.cc:1428] Setting key eda1b2f443d0c133cb8abe1e11b6255b
I20260619 14:36:27.247107 23486 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7d2129de5c347b587828bc4e5fe29e8: Neither blocks nor log segments found. Creating new log.
I20260619 14:36:27.248015 23486 log.cc:824] T 00000000000000000000000000000000 P e7d2129de5c347b587828bc4e5fe29e8: Log is configured to *not* fsync() on all Append() calls
I20260619 14:36:27.250427 23486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7d2129de5c347b587828bc4e5fe29e8: No bootstrap required, opened a new log
I20260619 14:36:27.254834 23486 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7d2129de5c347b587828bc4e5fe29e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7d2129de5c347b587828bc4e5fe29e8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32979 } }
I20260619 14:36:27.255131 23486 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7d2129de5c347b587828bc4e5fe29e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:36:27.255221 23486 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7d2129de5c347b587828bc4e5fe29e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7d2129de5c347b587828bc4e5fe29e8, State: Initialized, Role: FOLLOWER
I20260619 14:36:27.255743 23486 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7d2129de5c347b587828bc4e5fe29e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7d2129de5c347b587828bc4e5fe29e8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32979 } }
I20260619 14:36:27.255908 23486 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7d2129de5c347b587828bc4e5fe29e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:36:27.255966 23486 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7d2129de5c347b587828bc4e5fe29e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:36:27.256057 23486 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7d2129de5c347b587828bc4e5fe29e8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:36:27.256989 23489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:27.247693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:37365 (local address 127.0.0.1:32979)
0619 14:36:27.248193 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:27.248213 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:27.248259 (+ 46us) server_negotiation.cc:407] Connection header received
0619 14:36:27.249224 (+ 965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:27.249247 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:27.249698 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:27.250083 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:27.250552 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:27.252053 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:27.252954 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:27.253792 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:27.253964 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:27.254882 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:27.254888 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:27.254906 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:36:27.255431 (+ 525us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:27.255509 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:36:27.255538 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:27.255666 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:27.255840 (+ 174us) server_negotiation.cc:299] Negotiation successful
0619 14:36:27.255987 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":132,"threads_started":1}
I20260619 14:36:27.257251 23486 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7d2129de5c347b587828bc4e5fe29e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7d2129de5c347b587828bc4e5fe29e8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32979 } }
I20260619 14:36:27.257632 23486 leader_election.cc:302] T 00000000000000000000000000000000 P e7d2129de5c347b587828bc4e5fe29e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7d2129de5c347b587828bc4e5fe29e8; no voters:
I20260619 14:36:27.257994 23486 leader_election.cc:288] T 00000000000000000000000000000000 P e7d2129de5c347b587828bc4e5fe29e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:36:27.258064 23491 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7d2129de5c347b587828bc4e5fe29e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:36:27.258279 23491 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7d2129de5c347b587828bc4e5fe29e8 [term 1 LEADER]: Becoming Leader. State: Replica: e7d2129de5c347b587828bc4e5fe29e8, State: Running, Role: LEADER
I20260619 14:36:27.258574 23491 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7d2129de5c347b587828bc4e5fe29e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7d2129de5c347b587828bc4e5fe29e8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32979 } }
I20260619 14:36:27.259334 23486 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7d2129de5c347b587828bc4e5fe29e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:36:27.260170 2056 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:36:27.260236 23492 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7d2129de5c347b587828bc4e5fe29e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7d2129de5c347b587828bc4e5fe29e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7d2129de5c347b587828bc4e5fe29e8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32979 } } }
I20260619 14:36:27.260375 23492 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7d2129de5c347b587828bc4e5fe29e8 [sys.catalog]: This master's current role is: LEADER
I20260619 14:36:27.260697 23493 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7d2129de5c347b587828bc4e5fe29e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7d2129de5c347b587828bc4e5fe29e8. Latest consensus state: current_term: 1 leader_uuid: "e7d2129de5c347b587828bc4e5fe29e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7d2129de5c347b587828bc4e5fe29e8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32979 } } }
I20260619 14:36:27.260775 23493 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7d2129de5c347b587828bc4e5fe29e8 [sys.catalog]: This master's current role is: LEADER
I20260619 14:36:27.261736 23500 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:36:27.264994 23500 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:36:27.267261 23489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:27.262569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:52571 (local address 127.0.0.1:32979)
0619 14:36:27.262717 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:27.262723 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:27.262894 (+ 171us) server_negotiation.cc:407] Connection header received
0619 14:36:27.263010 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:27.263012 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:27.263053 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:27.263118 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:27.263434 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:27.264201 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:27.265281 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:27.265615 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:27.265738 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:27.266730 (+ 992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:27.266734 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:27.266737 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:36:27.266751 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:27.266782 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:36:27.266795 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:27.266890 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:27.267073 (+ 183us) server_negotiation.cc:299] Negotiation successful
0619 14:36:27.267112 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 14:36:27.271262 23500 catalog_manager.cc:1347] Generated new cluster ID: 8e2a1df4e30447cc8f57dd9e688a1c26
I20260619 14:36:27.271345 23500 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:36:27.285152 23500 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:36:27.286515 23500 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:36:27.309067 23500 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7d2129de5c347b587828bc4e5fe29e8: Generated new TSK 0
I20260619 14:36:27.309995 23500 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:36:27.316118 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 23418
W20260619 14:36:27.321496 23403 connection.cc:535] client connection to 127.0.0.1:32979 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32979): Connection reset by peer (error 104)
2026-06-19T14:36:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1408 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781879514993456-2056-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:36:27 dist-test-slave-70rm krb5kdc[23517](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:36:27 dist-test-slave-70rm krb5kdc[23517](info): set up 2 sockets
Jun 19 14:36:27 dist-test-slave-70rm krb5kdc[23517](info): commencing operation
krb5kdc: starting...
W20260619 14:36:28.342386 2056 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Jun 19 14:36:28 dist-test-slave-70rm krb5kdc[23517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:36:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:36:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781879514993456-2056-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781879514993456-2056-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:36:28.463378 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781879514993456-2056-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781879514993456-2056-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781879514993456-2056-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42423
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781879514993456-2056-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42423
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:36:28.572943 23530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:36:28.573217 23530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:36:28.573261 23530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:36:28.577275 23530 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:36:28.577338 23530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:36:28.577358 23530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:36:28.577374 23530 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:36:28.577389 23530 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:36:28.582769 23530 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781879514993456-2056-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781879514993456-2056-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42423
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781879514993456-2056-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781879514993456-2056-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:36:28.583868 23530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:36:28.584914 23530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:36:28.593302 23535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:28.593291 23538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:28.593291 23536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:28.593330 23530 server_base.cc:1034] running on GCE node
I20260619 14:36:28.594224 23530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:36:28.595407 23530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:36:28.596593 23530 hybrid_clock.cc:648] HybridClock initialized: now 1781879788596578 us; error 29 us; skew 500 ppm
Jun 19 14:36:28 dist-test-slave-70rm krb5kdc[23517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879788, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:36:28.604120 23530 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:36:28.605152 23530 webserver.cc:460] Webserver started at http://127.0.0.1:39323/ using document root <none> and password file <none>
I20260619 14:36:28.605757 23530 fs_manager.cc:359] Metadata directory not provided
I20260619 14:36:28.605824 23530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:36:28.606083 23530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:36:28.608268 23530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781879514993456-2056-0/minicluster-data/master-0/data/instance:
uuid: "2ea5b720fde6418cb1402f3ffa187b90"
format_stamp: "Formatted at 2026-06-19 14:36:28 on dist-test-slave-70rm"
server_key: "5e12503ba9bc23b6b2570e5aed60840d"
server_key_iv: "29194baa3ff9a50eac9f1f9d6bd937ec"
server_key_version: "encryptionkey@0"
I20260619 14:36:28.608876 23530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781879514993456-2056-0/minicluster-data/master-0/wal/instance:
uuid: "2ea5b720fde6418cb1402f3ffa187b90"
format_stamp: "Formatted at 2026-06-19 14:36:28 on dist-test-slave-70rm"
server_key: "5e12503ba9bc23b6b2570e5aed60840d"
server_key_iv: "29194baa3ff9a50eac9f1f9d6bd937ec"
server_key_version: "encryptionkey@0"
I20260619 14:36:28.612686 23530 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260619 14:36:28.615236 23545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:28.616693 23530 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:36:28.616813 23530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781879514993456-2056-0/minicluster-data/master-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781879514993456-2056-0/minicluster-data/master-0/wal
uuid: "2ea5b720fde6418cb1402f3ffa187b90"
format_stamp: "Formatted at 2026-06-19 14:36:28 on dist-test-slave-70rm"
server_key: "5e12503ba9bc23b6b2570e5aed60840d"
server_key_iv: "29194baa3ff9a50eac9f1f9d6bd937ec"
server_key_version: "encryptionkey@0"
I20260619 14:36:28.616995 23530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781879514993456-2056-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781879514993456-2056-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781879514993456-2056-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:36:28.622255 23530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:36:28.626097 23530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:36:28.626353 23530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:36:28.633694 23530 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42423
I20260619 14:36:28.633780 23597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42423 every 8 connection(s)
I20260619 14:36:28.634737 23530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
I20260619 14:36:28.637547 23598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:36:28.639513 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 23530
I20260619 14:36:28.639650 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781879514993456-2056-0/minicluster-data/master-0/wal/instance
I20260619 14:36:28.639945 2056 external_mini_cluster.cc:1428] Setting key 74387a118396099c987d2470c74aae27
I20260619 14:36:28.644687 23598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ea5b720fde6418cb1402f3ffa187b90: Bootstrap starting.
I20260619 14:36:28.645164 23601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:28.641253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:33713 (local address 127.0.0.1:42423)
0619 14:36:28.641673 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:28.641687 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:28.641725 (+ 38us) server_negotiation.cc:407] Connection header received
0619 14:36:28.642446 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:28.642466 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:28.642779 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:28.643131 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:28.643924 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:28.643928 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:28.643940 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:36:28.643950 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:28.644004 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:36:28.644027 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:28.644178 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:28.644246 (+ 68us) server_negotiation.cc:299] Negotiation successful
0619 14:36:28.644409 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":110,"threads_started":1}
I20260619 14:36:28.647171 23598 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ea5b720fde6418cb1402f3ffa187b90: Neither blocks nor log segments found. Creating new log.
I20260619 14:36:28.648031 23598 log.cc:824] T 00000000000000000000000000000000 P 2ea5b720fde6418cb1402f3ffa187b90: Log is configured to *not* fsync() on all Append() calls
I20260619 14:36:28.648211 2056 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:36:28.650362 23598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ea5b720fde6418cb1402f3ffa187b90: No bootstrap required, opened a new log
I20260619 14:36:28.652374 23601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:28.650573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:39361 (local address 127.0.0.1:42423)
0619 14:36:28.650699 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:28.650705 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:28.650875 (+ 170us) server_negotiation.cc:407] Connection header received
0619 14:36:28.651028 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:28.651033 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:28.651077 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:28.651160 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:28.651975 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:28.651979 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:28.651982 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:36:28.651994 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:28.652030 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:36:28.652041 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:28.652136 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:28.652208 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 14:36:28.652252 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 14:36:28.654472 23598 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ea5b720fde6418cb1402f3ffa187b90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ea5b720fde6418cb1402f3ffa187b90" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42423 } }
I20260619 14:36:28.654696 23598 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ea5b720fde6418cb1402f3ffa187b90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:36:28.654778 23598 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ea5b720fde6418cb1402f3ffa187b90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ea5b720fde6418cb1402f3ffa187b90, State: Initialized, Role: FOLLOWER
I20260619 14:36:28.655212 23598 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ea5b720fde6418cb1402f3ffa187b90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ea5b720fde6418cb1402f3ffa187b90" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42423 } }
I20260619 14:36:28.655328 23598 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ea5b720fde6418cb1402f3ffa187b90 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:36:28.655413 23598 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ea5b720fde6418cb1402f3ffa187b90 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:36:28.655543 23598 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ea5b720fde6418cb1402f3ffa187b90 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:36:28.656457 23598 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ea5b720fde6418cb1402f3ffa187b90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ea5b720fde6418cb1402f3ffa187b90" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42423 } }
I20260619 14:36:28.656802 23598 leader_election.cc:302] T 00000000000000000000000000000000 P 2ea5b720fde6418cb1402f3ffa187b90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ea5b720fde6418cb1402f3ffa187b90; no voters:
I20260619 14:36:28.657131 23598 leader_election.cc:288] T 00000000000000000000000000000000 P 2ea5b720fde6418cb1402f3ffa187b90 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:36:28.657269 23608 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ea5b720fde6418cb1402f3ffa187b90 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:36:28.657510 23608 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ea5b720fde6418cb1402f3ffa187b90 [term 1 LEADER]: Becoming Leader. State: Replica: 2ea5b720fde6418cb1402f3ffa187b90, State: Running, Role: LEADER
I20260619 14:36:28.657862 23608 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ea5b720fde6418cb1402f3ffa187b90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ea5b720fde6418cb1402f3ffa187b90" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42423 } }
I20260619 14:36:28.658535 23598 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ea5b720fde6418cb1402f3ffa187b90 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:36:28.659737 23610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ea5b720fde6418cb1402f3ffa187b90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ea5b720fde6418cb1402f3ffa187b90. Latest consensus state: current_term: 1 leader_uuid: "2ea5b720fde6418cb1402f3ffa187b90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ea5b720fde6418cb1402f3ffa187b90" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42423 } } }
I20260619 14:36:28.659861 23610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ea5b720fde6418cb1402f3ffa187b90 [sys.catalog]: This master's current role is: LEADER
I20260619 14:36:28.660224 23617 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:36:28.660212 23609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ea5b720fde6418cb1402f3ffa187b90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ea5b720fde6418cb1402f3ffa187b90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ea5b720fde6418cb1402f3ffa187b90" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42423 } } }
I20260619 14:36:28.660874 23609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ea5b720fde6418cb1402f3ffa187b90 [sys.catalog]: This master's current role is: LEADER
I20260619 14:36:28.663511 23617 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:36:28.669317 23617 catalog_manager.cc:1347] Generated new cluster ID: 000c46f4d83e414ba07a616c5587c013
I20260619 14:36:28.669391 23617 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:36:28.696250 23617 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:36:28.696694 23617 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:36:28.702452 23617 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ea5b720fde6418cb1402f3ffa187b90: Generated new TSK 0
I20260619 14:36:28.703286 23617 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:36:28.708782 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 23530
2026-06-19T14:36:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1392 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781879514993456-2056-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:36:28 dist-test-slave-70rm krb5kdc[23629](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:36:28 dist-test-slave-70rm krb5kdc[23629](info): set up 2 sockets
Jun 19 14:36:28 dist-test-slave-70rm krb5kdc[23629](info): commencing operation
krb5kdc: starting...
W20260619 14:36:29.762467 2056 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Jun 19 14:36:29 dist-test-slave-70rm krb5kdc[23629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:36:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:36:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.2.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab.
Entry for principal kudu/127.2.2.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab.
I20260619 14:36:29.887079 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781879514993456-2056-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781879514993456-2056-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781879514993456-2056-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:38865
--webserver_interface=127.2.2.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:40961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.2.62:38865
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:36:30.000288 23642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:36:30.000617 23642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:36:30.000690 23642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:36:30.004849 23642 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:36:30.004951 23642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:36:30.005020 23642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:36:30.005069 23642 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:36:30.005110 23642 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:36:30.010545 23642 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:40961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781879514993456-2056-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781879514993456-2056-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.2.62:38865
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:38865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.2.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781879514993456-2056-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:36:30.011634 23642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:36:30.012722 23642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:36:30.021095 23650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:30.021107 23647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:30.021109 23648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:30.021195 23642 server_base.cc:1034] running on GCE node
I20260619 14:36:30.021900 23642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:36:30.023097 23642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:36:30.024288 23642 hybrid_clock.cc:648] HybridClock initialized: now 1781879790024271 us; error 37 us; skew 500 ppm
Jun 19 14:36:30 dist-test-slave-70rm krb5kdc[23629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879790, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:36:30.027320 23642 init.cc:377] Logged in from keytab as kudu/127.2.2.62@KRBTEST.COM (short username kudu)
I20260619 14:36:30.028254 23642 webserver.cc:460] Webserver started at http://127.2.2.62:38151/ using document root <none> and password file <none>
I20260619 14:36:30.028784 23642 fs_manager.cc:359] Metadata directory not provided
I20260619 14:36:30.028846 23642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:36:30.029008 23642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:36:30.031118 23642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781879514993456-2056-0/minicluster-data/master-0/data/instance:
uuid: "8f07e3e13d5d4b03a0f5f810fa9d5e17"
format_stamp: "Formatted at 2026-06-19 14:36:30 on dist-test-slave-70rm"
server_key: "d40316b40fd64cab33cacd0ff9ec506b"
server_key_iv: "104a3bdd2439dae577ceaaee4c20e79a"
server_key_version: "encryptionkey@0"
I20260619 14:36:30.031629 23642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781879514993456-2056-0/minicluster-data/master-0/wal/instance:
uuid: "8f07e3e13d5d4b03a0f5f810fa9d5e17"
format_stamp: "Formatted at 2026-06-19 14:36:30 on dist-test-slave-70rm"
server_key: "d40316b40fd64cab33cacd0ff9ec506b"
server_key_iv: "104a3bdd2439dae577ceaaee4c20e79a"
server_key_version: "encryptionkey@0"
I20260619 14:36:30.035548 23642 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:36:30.038424 23657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:30.039745 23642 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:36:30.039857 23642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781879514993456-2056-0/minicluster-data/master-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781879514993456-2056-0/minicluster-data/master-0/wal
uuid: "8f07e3e13d5d4b03a0f5f810fa9d5e17"
format_stamp: "Formatted at 2026-06-19 14:36:30 on dist-test-slave-70rm"
server_key: "d40316b40fd64cab33cacd0ff9ec506b"
server_key_iv: "104a3bdd2439dae577ceaaee4c20e79a"
server_key_version: "encryptionkey@0"
I20260619 14:36:30.039955 23642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781879514993456-2056-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781879514993456-2056-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781879514993456-2056-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:36:30.051637 23642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:36:30.055228 23642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:36:30.055413 23642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:36:30.063148 23642 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.62:38865
I20260619 14:36:30.063164 23709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.62:38865 every 8 connection(s)
I20260619 14:36:30.064208 23642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
I20260619 14:36:30.067062 23710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:36:30.072608 23710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f07e3e13d5d4b03a0f5f810fa9d5e17: Bootstrap starting.
I20260619 14:36:30.073460 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 23642
I20260619 14:36:30.073606 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781879514993456-2056-0/minicluster-data/master-0/wal/instance
I20260619 14:36:30.073915 2056 external_mini_cluster.cc:1428] Setting key fe293c9e25fc668119e0e725d3c67a41
I20260619 14:36:30.075665 23710 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f07e3e13d5d4b03a0f5f810fa9d5e17: Neither blocks nor log segments found. Creating new log.
I20260619 14:36:30.076398 23710 log.cc:824] T 00000000000000000000000000000000 P 8f07e3e13d5d4b03a0f5f810fa9d5e17: Log is configured to *not* fsync() on all Append() calls
I20260619 14:36:30.078742 23710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f07e3e13d5d4b03a0f5f810fa9d5e17: No bootstrap required, opened a new log
Jun 19 14:36:30 dist-test-slave-70rm krb5kdc[23629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:36:30.083412 23710 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f07e3e13d5d4b03a0f5f810fa9d5e17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f07e3e13d5d4b03a0f5f810fa9d5e17" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 38865 } }
I20260619 14:36:30.083693 23710 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f07e3e13d5d4b03a0f5f810fa9d5e17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:36:30.083768 23710 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f07e3e13d5d4b03a0f5f810fa9d5e17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f07e3e13d5d4b03a0f5f810fa9d5e17, State: Initialized, Role: FOLLOWER
I20260619 14:36:30.084235 23710 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f07e3e13d5d4b03a0f5f810fa9d5e17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f07e3e13d5d4b03a0f5f810fa9d5e17" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 38865 } }
I20260619 14:36:30.084376 23710 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f07e3e13d5d4b03a0f5f810fa9d5e17 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:36:30.084456 23710 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f07e3e13d5d4b03a0f5f810fa9d5e17 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:36:30.084565 23710 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f07e3e13d5d4b03a0f5f810fa9d5e17 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:36:30.085441 23710 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f07e3e13d5d4b03a0f5f810fa9d5e17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f07e3e13d5d4b03a0f5f810fa9d5e17" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 38865 } }
I20260619 14:36:30.085783 23710 leader_election.cc:302] T 00000000000000000000000000000000 P 8f07e3e13d5d4b03a0f5f810fa9d5e17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f07e3e13d5d4b03a0f5f810fa9d5e17; no voters:
I20260619 14:36:30.086086 23710 leader_election.cc:288] T 00000000000000000000000000000000 P 8f07e3e13d5d4b03a0f5f810fa9d5e17 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:36:30.086246 23715 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f07e3e13d5d4b03a0f5f810fa9d5e17 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:36:30.086485 23715 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f07e3e13d5d4b03a0f5f810fa9d5e17 [term 1 LEADER]: Becoming Leader. State: Replica: 8f07e3e13d5d4b03a0f5f810fa9d5e17, State: Running, Role: LEADER
I20260619 14:36:30.086827 23715 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f07e3e13d5d4b03a0f5f810fa9d5e17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f07e3e13d5d4b03a0f5f810fa9d5e17" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 38865 } }
I20260619 14:36:30.087504 23710 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f07e3e13d5d4b03a0f5f810fa9d5e17 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:36:30.088054 23715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f07e3e13d5d4b03a0f5f810fa9d5e17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f07e3e13d5d4b03a0f5f810fa9d5e17. Latest consensus state: current_term: 1 leader_uuid: "8f07e3e13d5d4b03a0f5f810fa9d5e17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f07e3e13d5d4b03a0f5f810fa9d5e17" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 38865 } } }
I20260619 14:36:30.088195 23715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f07e3e13d5d4b03a0f5f810fa9d5e17 [sys.catalog]: This master's current role is: LEADER
I20260619 14:36:30.088464 23716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f07e3e13d5d4b03a0f5f810fa9d5e17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f07e3e13d5d4b03a0f5f810fa9d5e17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f07e3e13d5d4b03a0f5f810fa9d5e17" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 38865 } } }
I20260619 14:36:30.088557 23716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f07e3e13d5d4b03a0f5f810fa9d5e17 [sys.catalog]: This master's current role is: LEADER
I20260619 14:36:30.088814 23721 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:36:30.089679 23713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:30.075146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35075 (local address 127.2.2.62:38865)
0619 14:36:30.075592 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:30.075608 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:30.075644 (+ 36us) server_negotiation.cc:407] Connection header received
0619 14:36:30.076363 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:30.076380 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:30.076826 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:30.077217 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:30.078064 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:30.079199 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:30.080373 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:30.080884 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:30.081017 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:30.084078 (+ 3061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:30.084086 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:30.084110 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:36:30.084141 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:30.086016 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:30.086673 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:30.086679 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:30.086686 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:30.086822 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:30.087198 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:30.087201 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:30.087203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:30.087735 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:30.087961 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:30.088327 (+ 366us) server_negotiation.cc:299] Negotiation successful
0619 14:36:30.088590 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":122,"threads_started":1}
I20260619 14:36:30.092610 2056 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:36:30.092644 23721 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:36:30.099444 23721 catalog_manager.cc:1347] Generated new cluster ID: 544577d8787541c89638e5fd04982738
I20260619 14:36:30.099529 23721 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:36:30.102655 23713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:30.094739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57307 (local address 127.2.2.62:38865)
0619 14:36:30.094855 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:30.094859 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:30.095069 (+ 210us) server_negotiation.cc:407] Connection header received
0619 14:36:30.095197 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:30.095200 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:30.095242 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:30.095327 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:30.095953 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:30.096475 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:30.097274 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:30.097528 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:30.097631 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:30.099323 (+ 1692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:30.099328 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:30.099331 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:36:30.099357 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:30.100850 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:30.101337 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:30.101341 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:30.101342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:30.101423 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:30.101741 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:30.101744 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:30.101746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:30.101918 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:30.102092 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:30.102388 (+ 296us) server_negotiation.cc:299] Negotiation successful
0619 14:36:30.102484 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 14:36:30.114863 23721 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:36:30.116214 23721 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:36:30.126327 23721 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f07e3e13d5d4b03a0f5f810fa9d5e17: Generated new TSK 0
I20260619 14:36:30.127266 23721 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:36:30.136225 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 23642
W20260619 14:36:30.141685 23627 connection.cc:535] client connection to 127.2.2.62:38865 recv error: Network error: failed to read from TLS socket (remote: 127.2.2.62:38865): Connection reset by peer (error 104)
2026-06-19T14:36:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1427 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781879514993456-2056-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:36:30 dist-test-slave-70rm krb5kdc[23741](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:36:30 dist-test-slave-70rm krb5kdc[23741](info): set up 2 sockets
Jun 19 14:36:30 dist-test-slave-70rm krb5kdc[23741](info): commencing operation
krb5kdc: starting...
W20260619 14:36:31.170399 2056 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Jun 19 14:36:31 dist-test-slave-70rm krb5kdc[23741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:36:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:36:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.2.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab.
Entry for principal kudu/127.2.2.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab.
I20260619 14:36:31.293828 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781879514993456-2056-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781879514993456-2056-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781879514993456-2056-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:32927
--webserver_interface=127.2.2.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:38457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.2.62:32927
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:36:31.405650 23754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:36:31.406035 23754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:36:31.406158 23754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:36:31.410379 23754 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:36:31.410511 23754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:36:31.410576 23754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:36:31.410631 23754 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:36:31.410673 23754 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:36:31.416260 23754 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:38457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781879514993456-2056-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781879514993456-2056-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.2.62:32927
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:32927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.2.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23754
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781879514993456-2056-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:36:31.417423 23754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:36:31.418537 23754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:36:31.427191 23760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:31.427186 23759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:31.427191 23762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:31.427599 23754 server_base.cc:1034] running on GCE node
I20260619 14:36:31.428192 23754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:36:31.429381 23754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:36:31.430572 23754 hybrid_clock.cc:648] HybridClock initialized: now 1781879791430557 us; error 37 us; skew 500 ppm
Jun 19 14:36:31 dist-test-slave-70rm krb5kdc[23741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879791, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:36:31.433270 23754 init.cc:377] Logged in from keytab as kudu/127.2.2.62@KRBTEST.COM (short username kudu)
I20260619 14:36:31.434298 23754 webserver.cc:460] Webserver started at http://127.2.2.62:43907/ using document root <none> and password file <none>
I20260619 14:36:31.434875 23754 fs_manager.cc:359] Metadata directory not provided
I20260619 14:36:31.434963 23754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:36:31.435194 23754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:36:31.437422 23754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781879514993456-2056-0/minicluster-data/master-0/data/instance:
uuid: "1e6ae67134794dd79c3be356c9e62e7f"
format_stamp: "Formatted at 2026-06-19 14:36:31 on dist-test-slave-70rm"
server_key: "2fbb8f1f6d1df018d571a1a98d52a71c"
server_key_iv: "75eb63fbd221d6a6db1628af69b870e3"
server_key_version: "encryptionkey@0"
I20260619 14:36:31.438000 23754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781879514993456-2056-0/minicluster-data/master-0/wal/instance:
uuid: "1e6ae67134794dd79c3be356c9e62e7f"
format_stamp: "Formatted at 2026-06-19 14:36:31 on dist-test-slave-70rm"
server_key: "2fbb8f1f6d1df018d571a1a98d52a71c"
server_key_iv: "75eb63fbd221d6a6db1628af69b870e3"
server_key_version: "encryptionkey@0"
I20260619 14:36:31.441748 23754 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:36:31.444228 23769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:31.445451 23754 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:36:31.445591 23754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781879514993456-2056-0/minicluster-data/master-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781879514993456-2056-0/minicluster-data/master-0/wal
uuid: "1e6ae67134794dd79c3be356c9e62e7f"
format_stamp: "Formatted at 2026-06-19 14:36:31 on dist-test-slave-70rm"
server_key: "2fbb8f1f6d1df018d571a1a98d52a71c"
server_key_iv: "75eb63fbd221d6a6db1628af69b870e3"
server_key_version: "encryptionkey@0"
I20260619 14:36:31.445703 23754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781879514993456-2056-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781879514993456-2056-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781879514993456-2056-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:36:31.477357 23754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:36:31.480384 23754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:36:31.480592 23754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:36:31.487802 23754 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.62:32927
I20260619 14:36:31.487831 23821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.62:32927 every 8 connection(s)
I20260619 14:36:31.488855 23754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
I20260619 14:36:31.490352 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 23754
I20260619 14:36:31.490492 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781879514993456-2056-0/minicluster-data/master-0/wal/instance
I20260619 14:36:31.490780 2056 external_mini_cluster.cc:1428] Setting key 0591a5354737da32ff5b8b83a7788d36
I20260619 14:36:31.491948 23822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:36:31.497710 23822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e6ae67134794dd79c3be356c9e62e7f: Bootstrap starting.
I20260619 14:36:31.499656 23825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:31.492216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34653 (local address 127.2.2.62:32927)
0619 14:36:31.492643 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:31.492658 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:31.492693 (+ 35us) server_negotiation.cc:407] Connection header received
0619 14:36:31.493344 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:31.493361 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:31.493700 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:31.494049 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:31.494499 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:31.495465 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:31.496349 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:31.496834 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:31.497013 (+ 179us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:31.497950 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:31.497955 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:31.497967 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:36:31.498414 (+ 447us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:31.498470 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:36:31.498494 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:31.498630 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:31.498810 (+ 180us) server_negotiation.cc:299] Negotiation successful
0619 14:36:31.498947 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":144,"threads_started":1}
I20260619 14:36:31.500152 23822 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e6ae67134794dd79c3be356c9e62e7f: Neither blocks nor log segments found. Creating new log.
I20260619 14:36:31.500957 23822 log.cc:824] T 00000000000000000000000000000000 P 1e6ae67134794dd79c3be356c9e62e7f: Log is configured to *not* fsync() on all Append() calls
I20260619 14:36:31.502734 2056 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:36:31.503147 23822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e6ae67134794dd79c3be356c9e62e7f: No bootstrap required, opened a new log
I20260619 14:36:31.507884 23822 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e6ae67134794dd79c3be356c9e62e7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e6ae67134794dd79c3be356c9e62e7f" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 32927 } }
I20260619 14:36:31.508217 23822 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e6ae67134794dd79c3be356c9e62e7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:36:31.508296 23822 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e6ae67134794dd79c3be356c9e62e7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e6ae67134794dd79c3be356c9e62e7f, State: Initialized, Role: FOLLOWER
I20260619 14:36:31.508827 23822 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e6ae67134794dd79c3be356c9e62e7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e6ae67134794dd79c3be356c9e62e7f" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 32927 } }
I20260619 14:36:31.509057 23825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:31.504731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55531 (local address 127.2.2.62:32927)
0619 14:36:31.504840 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:31.504845 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:31.505080 (+ 235us) server_negotiation.cc:407] Connection header received
0619 14:36:31.505244 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:31.505246 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:31.505285 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:31.505353 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:31.505702 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:31.506270 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:31.507410 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:31.507627 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:31.507700 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:31.508592 (+ 892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:31.508596 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:31.508599 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:36:31.508612 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:31.508640 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:36:31.508653 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:31.508753 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:31.508880 (+ 127us) server_negotiation.cc:299] Negotiation successful
0619 14:36:31.508922 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 14:36:31.509063 23822 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e6ae67134794dd79c3be356c9e62e7f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:36:31.509177 23822 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e6ae67134794dd79c3be356c9e62e7f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:36:31.509275 23822 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e6ae67134794dd79c3be356c9e62e7f [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:36:31.510476 23822 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e6ae67134794dd79c3be356c9e62e7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e6ae67134794dd79c3be356c9e62e7f" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 32927 } }
I20260619 14:36:31.510816 23822 leader_election.cc:302] T 00000000000000000000000000000000 P 1e6ae67134794dd79c3be356c9e62e7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e6ae67134794dd79c3be356c9e62e7f; no voters:
I20260619 14:36:31.511142 23822 leader_election.cc:288] T 00000000000000000000000000000000 P 1e6ae67134794dd79c3be356c9e62e7f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:36:31.511248 23832 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e6ae67134794dd79c3be356c9e62e7f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:36:31.511493 23832 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e6ae67134794dd79c3be356c9e62e7f [term 1 LEADER]: Becoming Leader. State: Replica: 1e6ae67134794dd79c3be356c9e62e7f, State: Running, Role: LEADER
I20260619 14:36:31.511866 23832 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e6ae67134794dd79c3be356c9e62e7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e6ae67134794dd79c3be356c9e62e7f" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 32927 } }
I20260619 14:36:31.512372 23822 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e6ae67134794dd79c3be356c9e62e7f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:36:31.513556 23834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e6ae67134794dd79c3be356c9e62e7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e6ae67134794dd79c3be356c9e62e7f. Latest consensus state: current_term: 1 leader_uuid: "1e6ae67134794dd79c3be356c9e62e7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e6ae67134794dd79c3be356c9e62e7f" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 32927 } } }
I20260619 14:36:31.513696 23834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e6ae67134794dd79c3be356c9e62e7f [sys.catalog]: This master's current role is: LEADER
I20260619 14:36:31.513980 23833 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e6ae67134794dd79c3be356c9e62e7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e6ae67134794dd79c3be356c9e62e7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e6ae67134794dd79c3be356c9e62e7f" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 32927 } } }
I20260619 14:36:31.514082 23833 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e6ae67134794dd79c3be356c9e62e7f [sys.catalog]: This master's current role is: LEADER
W20260619 14:36:31.517264 23847 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1e6ae67134794dd79c3be356c9e62e7f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:36:31.517354 23847 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:36:31.517422 23841 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:36:31.518178 23841 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:36:31.523554 23841 catalog_manager.cc:1347] Generated new cluster ID: fda8ed54421248fc8ea4354ff71fe30e
I20260619 14:36:31.523635 23841 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:36:31.541078 23841 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:36:31.542876 23841 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:36:31.548442 23841 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e6ae67134794dd79c3be356c9e62e7f: Generated new TSK 0
I20260619 14:36:31.549247 23841 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:36:31.557511 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 23754
W20260619 14:36:31.562754 23739 connection.cc:535] client connection to 127.2.2.62:32927 recv error: Network error: failed to read from TLS socket (remote: 127.2.2.62:32927): Connection reset by peer (error 104)
2026-06-19T14:36:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1420 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781879514993456-2056-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:36:31 dist-test-slave-70rm krb5kdc[23853](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:36:31 dist-test-slave-70rm krb5kdc[23853](info): set up 2 sockets
Jun 19 14:36:31 dist-test-slave-70rm krb5kdc[23853](info): commencing operation
krb5kdc: starting...
W20260619 14:36:32.602442 2056 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Jun 19 14:36:32 dist-test-slave-70rm krb5kdc[23853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:36:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:36:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.2.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab.
Entry for principal kudu/127.2.2.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab.
I20260619 14:36:32.724808 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781879514993456-2056-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781879514993456-2056-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781879514993456-2056-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:41289
--webserver_interface=127.2.2.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:35417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.2.62:41289
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:36:32.836321 23866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:36:32.836617 23866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:36:32.836660 23866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:36:32.840775 23866 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:36:32.840840 23866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:36:32.840860 23866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:36:32.840878 23866 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:36:32.840893 23866 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:36:32.846467 23866 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:35417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781879514993456-2056-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781879514993456-2056-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.2.62:41289
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:41289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.2.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781879514993456-2056-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:36:32.847546 23866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:36:32.848551 23866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:36:32.857501 23871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:32.857481 23872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:32.857563 23866 server_base.cc:1034] running on GCE node
W20260619 14:36:32.857504 23874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:32.858361 23866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:36:32.859582 23866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:36:32.860759 23866 hybrid_clock.cc:648] HybridClock initialized: now 1781879792860744 us; error 29 us; skew 500 ppm
Jun 19 14:36:32 dist-test-slave-70rm krb5kdc[23853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879792, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:36:32.863739 23866 init.cc:377] Logged in from keytab as kudu/127.2.2.62@KRBTEST.COM (short username kudu)
I20260619 14:36:32.864715 23866 webserver.cc:460] Webserver started at http://127.2.2.62:39679/ using document root <none> and password file <none>
I20260619 14:36:32.865332 23866 fs_manager.cc:359] Metadata directory not provided
I20260619 14:36:32.865432 23866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:36:32.865686 23866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:36:32.867980 23866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781879514993456-2056-0/minicluster-data/master-0/data/instance:
uuid: "2724ce40b88b44a1ba94ee14c26da3d4"
format_stamp: "Formatted at 2026-06-19 14:36:32 on dist-test-slave-70rm"
server_key: "e58be3f17843351d3ceb7483416bec98"
server_key_iv: "d68b13498baf9f1afa7bc5eeedaf36ec"
server_key_version: "encryptionkey@0"
I20260619 14:36:32.868600 23866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781879514993456-2056-0/minicluster-data/master-0/wal/instance:
uuid: "2724ce40b88b44a1ba94ee14c26da3d4"
format_stamp: "Formatted at 2026-06-19 14:36:32 on dist-test-slave-70rm"
server_key: "e58be3f17843351d3ceb7483416bec98"
server_key_iv: "d68b13498baf9f1afa7bc5eeedaf36ec"
server_key_version: "encryptionkey@0"
I20260619 14:36:32.872838 23866 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260619 14:36:32.875541 23881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:32.876901 23866 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:36:32.877063 23866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781879514993456-2056-0/minicluster-data/master-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781879514993456-2056-0/minicluster-data/master-0/wal
uuid: "2724ce40b88b44a1ba94ee14c26da3d4"
format_stamp: "Formatted at 2026-06-19 14:36:32 on dist-test-slave-70rm"
server_key: "e58be3f17843351d3ceb7483416bec98"
server_key_iv: "d68b13498baf9f1afa7bc5eeedaf36ec"
server_key_version: "encryptionkey@0"
I20260619 14:36:32.877206 23866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781879514993456-2056-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781879514993456-2056-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781879514993456-2056-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:36:32.882258 23866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:36:32.885426 23866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:36:32.885653 23866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:36:32.892649 23866 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.62:41289
I20260619 14:36:32.892706 23933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.62:41289 every 8 connection(s)
I20260619 14:36:32.893662 23866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
I20260619 14:36:32.896474 23934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:36:32.901121 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 23866
I20260619 14:36:32.901283 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781879514993456-2056-0/minicluster-data/master-0/wal/instance
I20260619 14:36:32.901557 2056 external_mini_cluster.cc:1428] Setting key cfa1c9db52691f3716c15ea96b41c6b2
I20260619 14:36:32.902261 23934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2724ce40b88b44a1ba94ee14c26da3d4: Bootstrap starting.
I20260619 14:36:32.905059 23934 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2724ce40b88b44a1ba94ee14c26da3d4: Neither blocks nor log segments found. Creating new log.
I20260619 14:36:32.905977 23934 log.cc:824] T 00000000000000000000000000000000 P 2724ce40b88b44a1ba94ee14c26da3d4: Log is configured to *not* fsync() on all Append() calls
I20260619 14:36:32.907281 23937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:32.902930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52379 (local address 127.2.2.62:41289)
0619 14:36:32.903370 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:32.903389 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:32.903438 (+ 49us) server_negotiation.cc:407] Connection header received
0619 14:36:32.904298 (+ 860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:32.904323 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:32.904703 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:32.905089 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:32.905855 (+ 766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:32.905863 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:32.905881 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:36:32.905892 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:32.905954 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:36:32.905981 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:32.906090 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:32.906271 (+ 181us) server_negotiation.cc:299] Negotiation successful
0619 14:36:32.906434 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":112,"threads_started":1}
I20260619 14:36:32.908236 23934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2724ce40b88b44a1ba94ee14c26da3d4: No bootstrap required, opened a new log
I20260619 14:36:32.910667 2056 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:36:32.912519 23934 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2724ce40b88b44a1ba94ee14c26da3d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2724ce40b88b44a1ba94ee14c26da3d4" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41289 } }
I20260619 14:36:32.912846 23934 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2724ce40b88b44a1ba94ee14c26da3d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:36:32.912904 23934 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2724ce40b88b44a1ba94ee14c26da3d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2724ce40b88b44a1ba94ee14c26da3d4, State: Initialized, Role: FOLLOWER
I20260619 14:36:32.913481 23934 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2724ce40b88b44a1ba94ee14c26da3d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2724ce40b88b44a1ba94ee14c26da3d4" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41289 } }
I20260619 14:36:32.913650 23934 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2724ce40b88b44a1ba94ee14c26da3d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:36:32.913718 23934 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2724ce40b88b44a1ba94ee14c26da3d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:36:32.913822 23934 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2724ce40b88b44a1ba94ee14c26da3d4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:36:32.914525 23937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:32.912728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43387 (local address 127.2.2.62:41289)
0619 14:36:32.912847 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:32.912852 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:32.913068 (+ 216us) server_negotiation.cc:407] Connection header received
0619 14:36:32.913213 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:32.913216 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:32.913258 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:32.913338 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:32.914104 (+ 766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:32.914108 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:32.914110 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:36:32.914154 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:32.914185 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:36:32.914203 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:32.914303 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:32.914362 (+ 59us) server_negotiation.cc:299] Negotiation successful
0619 14:36:32.914401 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 14:36:32.915170 23934 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2724ce40b88b44a1ba94ee14c26da3d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2724ce40b88b44a1ba94ee14c26da3d4" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41289 } }
I20260619 14:36:32.915552 23934 leader_election.cc:302] T 00000000000000000000000000000000 P 2724ce40b88b44a1ba94ee14c26da3d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2724ce40b88b44a1ba94ee14c26da3d4; no voters:
I20260619 14:36:32.915903 23934 leader_election.cc:288] T 00000000000000000000000000000000 P 2724ce40b88b44a1ba94ee14c26da3d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:36:32.916041 23944 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2724ce40b88b44a1ba94ee14c26da3d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:36:32.916287 23944 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2724ce40b88b44a1ba94ee14c26da3d4 [term 1 LEADER]: Becoming Leader. State: Replica: 2724ce40b88b44a1ba94ee14c26da3d4, State: Running, Role: LEADER
I20260619 14:36:32.916612 23944 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2724ce40b88b44a1ba94ee14c26da3d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2724ce40b88b44a1ba94ee14c26da3d4" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41289 } }
I20260619 14:36:32.917179 23934 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2724ce40b88b44a1ba94ee14c26da3d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:36:32.918412 23945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2724ce40b88b44a1ba94ee14c26da3d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2724ce40b88b44a1ba94ee14c26da3d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2724ce40b88b44a1ba94ee14c26da3d4" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41289 } } }
I20260619 14:36:32.918567 23945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2724ce40b88b44a1ba94ee14c26da3d4 [sys.catalog]: This master's current role is: LEADER
I20260619 14:36:32.918412 23946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2724ce40b88b44a1ba94ee14c26da3d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2724ce40b88b44a1ba94ee14c26da3d4. Latest consensus state: current_term: 1 leader_uuid: "2724ce40b88b44a1ba94ee14c26da3d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2724ce40b88b44a1ba94ee14c26da3d4" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41289 } } }
I20260619 14:36:32.918666 23946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2724ce40b88b44a1ba94ee14c26da3d4 [sys.catalog]: This master's current role is: LEADER
W20260619 14:36:32.922104 23959 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2724ce40b88b44a1ba94ee14c26da3d4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:36:32.922258 23959 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:36:32.922331 23953 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:36:32.923046 23953 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:36:32.928316 23953 catalog_manager.cc:1347] Generated new cluster ID: 8052d00f91cc4c1995b749e32a777d9c
I20260619 14:36:32.928390 23953 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:36:32.969544 23953 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:36:32.969897 23953 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:36:32.976523 23953 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2724ce40b88b44a1ba94ee14c26da3d4: Generated new TSK 0
I20260619 14:36:32.977275 23953 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:36:32.987160 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 23866
2026-06-19T14:36:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1430 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781879514993456-2056-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:36:33 dist-test-slave-70rm krb5kdc[23965](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:36:33 dist-test-slave-70rm krb5kdc[23965](info): set up 2 sockets
Jun 19 14:36:33 dist-test-slave-70rm krb5kdc[23965](info): commencing operation
krb5kdc: starting...
W20260619 14:36:34.026410 2056 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Jun 19 14:36:34 dist-test-slave-70rm krb5kdc[23965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:36:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:36:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.2.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab.
Entry for principal kudu/127.2.2.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab.
I20260619 14:36:34.148590 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781879514993456-2056-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781879514993456-2056-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781879514993456-2056-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:44729
--webserver_interface=127.2.2.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:43389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.2.62:44729
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:36:34.259379 23978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:36:34.259655 23978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:36:34.259732 23978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:36:34.263736 23978 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:36:34.263794 23978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:36:34.263813 23978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:36:34.263830 23978 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:36:34.263875 23978 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:36:34.269176 23978 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:43389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781879514993456-2056-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781879514993456-2056-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.2.62:44729
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:44729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.2.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781879514993456-2056-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:36:34.270287 23978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:36:34.271322 23978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:36:34.279999 23983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:34.279999 23984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:34.280038 23978 server_base.cc:1034] running on GCE node
W20260619 14:36:34.279992 23986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:34.280851 23978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:36:34.281930 23978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:36:34.283107 23978 hybrid_clock.cc:648] HybridClock initialized: now 1781879794283096 us; error 55 us; skew 500 ppm
Jun 19 14:36:34 dist-test-slave-70rm krb5kdc[23965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879794, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:36:34.291749 23978 init.cc:377] Logged in from keytab as kudu/127.2.2.62@KRBTEST.COM (short username kudu)
I20260619 14:36:34.292665 23978 webserver.cc:460] Webserver started at http://127.2.2.62:38891/ using document root <none> and password file <none>
I20260619 14:36:34.293192 23978 fs_manager.cc:359] Metadata directory not provided
I20260619 14:36:34.293252 23978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:36:34.293432 23978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:36:34.295573 23978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781879514993456-2056-0/minicluster-data/master-0/data/instance:
uuid: "d4a7ee8930374ecc80fe5b598fcb0cc8"
format_stamp: "Formatted at 2026-06-19 14:36:34 on dist-test-slave-70rm"
server_key: "43246eb5abb3e3a5596de6fff0eedff7"
server_key_iv: "857e9d3b385af22498d2a67f385c3e07"
server_key_version: "encryptionkey@0"
I20260619 14:36:34.296087 23978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781879514993456-2056-0/minicluster-data/master-0/wal/instance:
uuid: "d4a7ee8930374ecc80fe5b598fcb0cc8"
format_stamp: "Formatted at 2026-06-19 14:36:34 on dist-test-slave-70rm"
server_key: "43246eb5abb3e3a5596de6fff0eedff7"
server_key_iv: "857e9d3b385af22498d2a67f385c3e07"
server_key_version: "encryptionkey@0"
I20260619 14:36:34.299839 23978 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260619 14:36:34.302277 23993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:34.303593 23978 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:36:34.303727 23978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781879514993456-2056-0/minicluster-data/master-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781879514993456-2056-0/minicluster-data/master-0/wal
uuid: "d4a7ee8930374ecc80fe5b598fcb0cc8"
format_stamp: "Formatted at 2026-06-19 14:36:34 on dist-test-slave-70rm"
server_key: "43246eb5abb3e3a5596de6fff0eedff7"
server_key_iv: "857e9d3b385af22498d2a67f385c3e07"
server_key_version: "encryptionkey@0"
I20260619 14:36:34.303833 23978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781879514993456-2056-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781879514993456-2056-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781879514993456-2056-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:36:34.332316 23978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:36:34.335739 23978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:36:34.335945 23978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:36:34.342723 23978 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.62:44729
I20260619 14:36:34.342744 24045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.62:44729 every 8 connection(s)
I20260619 14:36:34.343729 23978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
I20260619 14:36:34.345366 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 23978
I20260619 14:36:34.345516 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781879514993456-2056-0/minicluster-data/master-0/wal/instance
I20260619 14:36:34.345793 2056 external_mini_cluster.cc:1428] Setting key 690e449f8199c98f7347ccd5dac4f5dd
I20260619 14:36:34.346892 24046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:36:34.353543 24046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4a7ee8930374ecc80fe5b598fcb0cc8: Bootstrap starting.
Jun 19 14:36:34 dist-test-slave-70rm krb5kdc[23965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:36:34.355929 24046 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4a7ee8930374ecc80fe5b598fcb0cc8: Neither blocks nor log segments found. Creating new log.
I20260619 14:36:34.356904 24046 log.cc:824] T 00000000000000000000000000000000 P d4a7ee8930374ecc80fe5b598fcb0cc8: Log is configured to *not* fsync() on all Append() calls
I20260619 14:36:34.359202 24046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4a7ee8930374ecc80fe5b598fcb0cc8: No bootstrap required, opened a new log
I20260619 14:36:34.360610 24049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:34.347224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39271 (local address 127.2.2.62:44729)
0619 14:36:34.347632 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:34.347651 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:34.347702 (+ 51us) server_negotiation.cc:407] Connection header received
0619 14:36:34.348536 (+ 834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:34.348558 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:34.349003 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:34.349371 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:34.350046 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:34.351516 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:34.352341 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:34.353013 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:34.353174 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:34.355637 (+ 2463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:34.355644 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:34.355660 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:36:34.355701 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:34.357599 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:34.358036 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:34.358041 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:34.358047 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:34.358199 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:34.358446 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:34.358449 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:34.358451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:34.358953 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:34.359131 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:34.359393 (+ 262us) server_negotiation.cc:299] Negotiation successful
0619 14:36:34.359707 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":120,"threads_started":1}
I20260619 14:36:34.363665 2056 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:36:34.363613 24046 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4a7ee8930374ecc80fe5b598fcb0cc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4a7ee8930374ecc80fe5b598fcb0cc8" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 44729 } }
I20260619 14:36:34.363867 24046 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4a7ee8930374ecc80fe5b598fcb0cc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:36:34.363963 24046 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4a7ee8930374ecc80fe5b598fcb0cc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4a7ee8930374ecc80fe5b598fcb0cc8, State: Initialized, Role: FOLLOWER
I20260619 14:36:34.364408 24046 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4a7ee8930374ecc80fe5b598fcb0cc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4a7ee8930374ecc80fe5b598fcb0cc8" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 44729 } }
I20260619 14:36:34.364528 24046 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4a7ee8930374ecc80fe5b598fcb0cc8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:36:34.364601 24046 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4a7ee8930374ecc80fe5b598fcb0cc8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:36:34.364697 24046 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4a7ee8930374ecc80fe5b598fcb0cc8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:36:34.365775 24046 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4a7ee8930374ecc80fe5b598fcb0cc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4a7ee8930374ecc80fe5b598fcb0cc8" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 44729 } }
I20260619 14:36:34.366242 24046 leader_election.cc:302] T 00000000000000000000000000000000 P d4a7ee8930374ecc80fe5b598fcb0cc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4a7ee8930374ecc80fe5b598fcb0cc8; no voters:
I20260619 14:36:34.366626 24046 leader_election.cc:288] T 00000000000000000000000000000000 P d4a7ee8930374ecc80fe5b598fcb0cc8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:36:34.366724 24056 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4a7ee8930374ecc80fe5b598fcb0cc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:36:34.366959 24056 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4a7ee8930374ecc80fe5b598fcb0cc8 [term 1 LEADER]: Becoming Leader. State: Replica: d4a7ee8930374ecc80fe5b598fcb0cc8, State: Running, Role: LEADER
I20260619 14:36:34.367314 24056 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4a7ee8930374ecc80fe5b598fcb0cc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4a7ee8930374ecc80fe5b598fcb0cc8" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 44729 } }
I20260619 14:36:34.367849 24046 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4a7ee8930374ecc80fe5b598fcb0cc8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:36:34.369009 24057 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4a7ee8930374ecc80fe5b598fcb0cc8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4a7ee8930374ecc80fe5b598fcb0cc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4a7ee8930374ecc80fe5b598fcb0cc8" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 44729 } } }
I20260619 14:36:34.369205 24057 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4a7ee8930374ecc80fe5b598fcb0cc8 [sys.catalog]: This master's current role is: LEADER
I20260619 14:36:34.369475 24057 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4a7ee8930374ecc80fe5b598fcb0cc8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4a7ee8930374ecc80fe5b598fcb0cc8. Latest consensus state: current_term: 1 leader_uuid: "d4a7ee8930374ecc80fe5b598fcb0cc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4a7ee8930374ecc80fe5b598fcb0cc8" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 44729 } } }
I20260619 14:36:34.369629 24057 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4a7ee8930374ecc80fe5b598fcb0cc8 [sys.catalog]: This master's current role is: LEADER
I20260619 14:36:34.370635 24066 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:36:34.373924 24066 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:36:34.374938 24049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:34.365832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46051 (local address 127.2.2.62:44729)
0619 14:36:34.365952 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:34.365956 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:34.366079 (+ 123us) server_negotiation.cc:407] Connection header received
0619 14:36:34.366216 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:34.366218 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:34.366258 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:34.366330 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:34.367181 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:34.367964 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:34.369190 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:34.369578 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:34.369660 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:34.371457 (+ 1797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:34.371461 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:34.371464 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:36:34.371489 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:34.373130 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:34.373605 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:34.373609 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:34.373611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:34.373699 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:34.373993 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:34.373997 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:34.373998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:34.374254 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:34.374380 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:34.374651 (+ 271us) server_negotiation.cc:299] Negotiation successful
0619 14:36:34.374779 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 14:36:34.379549 24066 catalog_manager.cc:1347] Generated new cluster ID: 9f2fd0c8a2d04f048b715773cd6c41c9
I20260619 14:36:34.379629 24066 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:36:34.390962 24066 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:36:34.391928 24066 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:36:34.401584 24066 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4a7ee8930374ecc80fe5b598fcb0cc8: Generated new TSK 0
I20260619 14:36:34.402397 24066 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:36:34.409827 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 23978
2026-06-19T14:36:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1422 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781879514993456-2056-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:36:34 dist-test-slave-70rm krb5kdc[24077](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:36:34 dist-test-slave-70rm krb5kdc[24077](info): set up 2 sockets
Jun 19 14:36:34 dist-test-slave-70rm krb5kdc[24077](info): commencing operation
krb5kdc: starting...
W20260619 14:36:35.438607 2056 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Jun 19 14:36:35 dist-test-slave-70rm krb5kdc[24077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:36:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:36:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.2.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab.
Entry for principal kudu/127.2.2.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab.
I20260619 14:36:35.556891 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781879514993456-2056-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781879514993456-2056-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781879514993456-2056-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:39563
--webserver_interface=127.2.2.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:40759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.2.62:39563
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:36:35.670941 24090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:36:35.671219 24090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:36:35.671262 24090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:36:35.675189 24090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:36:35.675244 24090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:36:35.675263 24090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:36:35.675279 24090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:36:35.675295 24090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:36:35.680426 24090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:40759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781879514993456-2056-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781879514993456-2056-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.2.62:39563
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:39563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.2.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781879514993456-2056-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:36:35.681428 24090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:36:35.682610 24090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:36:35.691135 24098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:35.691135 24096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:35.691174 24095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:35.691390 24090 server_base.cc:1034] running on GCE node
I20260619 14:36:35.692042 24090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:36:35.693208 24090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:36:35.694418 24090 hybrid_clock.cc:648] HybridClock initialized: now 1781879795694401 us; error 31 us; skew 500 ppm
Jun 19 14:36:35 dist-test-slave-70rm krb5kdc[24077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879795, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:36:35.697117 24090 init.cc:377] Logged in from keytab as kudu/127.2.2.62@KRBTEST.COM (short username kudu)
I20260619 14:36:35.698037 24090 webserver.cc:460] Webserver started at http://127.2.2.62:36627/ using document root <none> and password file <none>
I20260619 14:36:35.698613 24090 fs_manager.cc:359] Metadata directory not provided
I20260619 14:36:35.698705 24090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:36:35.698921 24090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:36:35.700954 24090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781879514993456-2056-0/minicluster-data/master-0/data/instance:
uuid: "0d8be5357acd40669305120b134f7e02"
format_stamp: "Formatted at 2026-06-19 14:36:35 on dist-test-slave-70rm"
server_key: "77c62ed0fcf95751dcc0d8646c664162"
server_key_iv: "d08307f8b3de2851d65afc5ff73bd5ce"
server_key_version: "encryptionkey@0"
I20260619 14:36:35.701498 24090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781879514993456-2056-0/minicluster-data/master-0/wal/instance:
uuid: "0d8be5357acd40669305120b134f7e02"
format_stamp: "Formatted at 2026-06-19 14:36:35 on dist-test-slave-70rm"
server_key: "77c62ed0fcf95751dcc0d8646c664162"
server_key_iv: "d08307f8b3de2851d65afc5ff73bd5ce"
server_key_version: "encryptionkey@0"
I20260619 14:36:35.705116 24090 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 14:36:35.707638 24105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:35.708875 24090 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:36:35.709018 24090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781879514993456-2056-0/minicluster-data/master-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781879514993456-2056-0/minicluster-data/master-0/wal
uuid: "0d8be5357acd40669305120b134f7e02"
format_stamp: "Formatted at 2026-06-19 14:36:35 on dist-test-slave-70rm"
server_key: "77c62ed0fcf95751dcc0d8646c664162"
server_key_iv: "d08307f8b3de2851d65afc5ff73bd5ce"
server_key_version: "encryptionkey@0"
I20260619 14:36:35.709129 24090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781879514993456-2056-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781879514993456-2056-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781879514993456-2056-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:36:35.731791 24090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:36:35.734931 24090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:36:35.735142 24090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:36:35.742416 24090 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.62:39563
I20260619 14:36:35.742501 24157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.62:39563 every 8 connection(s)
I20260619 14:36:35.743435 24090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
I20260619 14:36:35.746317 24158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:36:35.751950 24158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d8be5357acd40669305120b134f7e02: Bootstrap starting.
I20260619 14:36:35.753330 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 24090
I20260619 14:36:35.753459 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781879514993456-2056-0/minicluster-data/master-0/wal/instance
I20260619 14:36:35.753755 2056 external_mini_cluster.cc:1428] Setting key 5dec04fad6d37d7bf6eaf24e464c6b48
I20260619 14:36:35.754895 24158 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d8be5357acd40669305120b134f7e02: Neither blocks nor log segments found. Creating new log.
I20260619 14:36:35.755662 24158 log.cc:824] T 00000000000000000000000000000000 P 0d8be5357acd40669305120b134f7e02: Log is configured to *not* fsync() on all Append() calls
I20260619 14:36:35.757762 24158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d8be5357acd40669305120b134f7e02: No bootstrap required, opened a new log
I20260619 14:36:35.762214 24158 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d8be5357acd40669305120b134f7e02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8be5357acd40669305120b134f7e02" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 39563 } }
I20260619 14:36:35.762475 24158 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d8be5357acd40669305120b134f7e02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:36:35.762560 24158 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d8be5357acd40669305120b134f7e02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d8be5357acd40669305120b134f7e02, State: Initialized, Role: FOLLOWER
I20260619 14:36:35.763039 24161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:35.754921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60043 (local address 127.2.2.62:39563)
0619 14:36:35.755360 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:35.755375 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:35.755409 (+ 34us) server_negotiation.cc:407] Connection header received
0619 14:36:35.756179 (+ 770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:35.756204 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:35.756604 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:35.756892 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:35.757221 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:35.758267 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:35.759504 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:35.760031 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:35.760163 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:35.761081 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:35.761087 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:35.761101 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:36:35.761520 (+ 419us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:35.761581 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:36:35.761603 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:35.761683 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:35.761848 (+ 165us) server_negotiation.cc:299] Negotiation successful
0619 14:36:35.762086 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":136,"threads_started":1}
I20260619 14:36:35.763003 24158 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d8be5357acd40669305120b134f7e02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8be5357acd40669305120b134f7e02" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 39563 } }
I20260619 14:36:35.763170 24158 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d8be5357acd40669305120b134f7e02 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:36:35.763268 24158 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d8be5357acd40669305120b134f7e02 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:36:35.763373 24158 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d8be5357acd40669305120b134f7e02 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:36:35.764474 24158 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d8be5357acd40669305120b134f7e02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8be5357acd40669305120b134f7e02" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 39563 } }
I20260619 14:36:35.764842 24158 leader_election.cc:302] T 00000000000000000000000000000000 P 0d8be5357acd40669305120b134f7e02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d8be5357acd40669305120b134f7e02; no voters:
I20260619 14:36:35.765200 24158 leader_election.cc:288] T 00000000000000000000000000000000 P 0d8be5357acd40669305120b134f7e02 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:36:35.765301 24163 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d8be5357acd40669305120b134f7e02 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:36:35.765542 24163 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d8be5357acd40669305120b134f7e02 [term 1 LEADER]: Becoming Leader. State: Replica: 0d8be5357acd40669305120b134f7e02, State: Running, Role: LEADER
I20260619 14:36:35.765918 24163 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d8be5357acd40669305120b134f7e02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8be5357acd40669305120b134f7e02" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 39563 } }
I20260619 14:36:35.766072 2056 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:36:35.766592 24158 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d8be5357acd40669305120b134f7e02 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:36:35.767450 24165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d8be5357acd40669305120b134f7e02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d8be5357acd40669305120b134f7e02. Latest consensus state: current_term: 1 leader_uuid: "0d8be5357acd40669305120b134f7e02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8be5357acd40669305120b134f7e02" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 39563 } } }
I20260619 14:36:35.767580 24165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d8be5357acd40669305120b134f7e02 [sys.catalog]: This master's current role is: LEADER
I20260619 14:36:35.767937 24164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d8be5357acd40669305120b134f7e02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d8be5357acd40669305120b134f7e02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8be5357acd40669305120b134f7e02" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 39563 } } }
I20260619 14:36:35.768011 24164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d8be5357acd40669305120b134f7e02 [sys.catalog]: This master's current role is: LEADER
I20260619 14:36:35.768314 24176 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:36:35.771565 24176 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:36:35.773911 24161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:35.769217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35277 (local address 127.2.2.62:39563)
0619 14:36:35.769329 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:35.769333 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:35.769504 (+ 171us) server_negotiation.cc:407] Connection header received
0619 14:36:35.769697 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:35.769700 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:35.769748 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:35.769832 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:35.770138 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:35.770857 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:35.772022 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:35.772341 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:35.772442 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:35.773434 (+ 992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:35.773438 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:35.773441 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:36:35.773455 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:35.773488 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:36:35.773503 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:35.773594 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:35.773702 (+ 108us) server_negotiation.cc:299] Negotiation successful
0619 14:36:35.773760 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 14:36:35.778282 24176 catalog_manager.cc:1347] Generated new cluster ID: aa99983fd94145598be8772d636eddc3
I20260619 14:36:35.778355 24176 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:36:35.786419 24176 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:36:35.787385 24176 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:36:35.800611 24176 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d8be5357acd40669305120b134f7e02: Generated new TSK 0
I20260619 14:36:35.801386 24176 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:36:35.811465 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 24090
2026-06-19T14:36:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1401 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781879514993456-2056-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:36:35 dist-test-slave-70rm krb5kdc[24189](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:36:35 dist-test-slave-70rm krb5kdc[24189](info): set up 2 sockets
Jun 19 14:36:35 dist-test-slave-70rm krb5kdc[24189](info): commencing operation
krb5kdc: starting...
W20260619 14:36:36.838832 2056 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Jun 19 14:36:36 dist-test-slave-70rm krb5kdc[24189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:36:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:36:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.2.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab.
Entry for principal kudu/127.2.2.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab.
I20260619 14:36:36.959064 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781879514993456-2056-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781879514993456-2056-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781879514993456-2056-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:43147
--webserver_interface=127.2.2.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:46741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.2.62:43147
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:36:37.068498 24202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:36:37.068831 24202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:36:37.068923 24202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:36:37.072911 24202 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:36:37.073002 24202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:36:37.073041 24202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:36:37.073087 24202 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:36:37.073122 24202 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:36:37.078490 24202 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:46741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781879514993456-2056-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781879514993456-2056-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.2.62:43147
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:43147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.2.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781879514993456-2056-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:36:37.079608 24202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:36:37.080651 24202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:36:37.089114 24210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:37.089121 24208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:37.089184 24202 server_base.cc:1034] running on GCE node
W20260619 14:36:37.089250 24207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:37.089926 24202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:36:37.091143 24202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:36:37.092329 24202 hybrid_clock.cc:648] HybridClock initialized: now 1781879797092297 us; error 44 us; skew 500 ppm
Jun 19 14:36:37 dist-test-slave-70rm krb5kdc[24189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879797, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:36:37.095129 24202 init.cc:377] Logged in from keytab as kudu/127.2.2.62@KRBTEST.COM (short username kudu)
I20260619 14:36:37.095991 24202 webserver.cc:460] Webserver started at http://127.2.2.62:34029/ using document root <none> and password file <none>
I20260619 14:36:37.096547 24202 fs_manager.cc:359] Metadata directory not provided
I20260619 14:36:37.096635 24202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:36:37.096863 24202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:36:37.099001 24202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781879514993456-2056-0/minicluster-data/master-0/data/instance:
uuid: "e1aa938182db4af6a4deb259a5281368"
format_stamp: "Formatted at 2026-06-19 14:36:37 on dist-test-slave-70rm"
server_key: "e297ca62cc41f0531b1c31da48631397"
server_key_iv: "c36b63aaa19880c925da637563ad795e"
server_key_version: "encryptionkey@0"
I20260619 14:36:37.099582 24202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781879514993456-2056-0/minicluster-data/master-0/wal/instance:
uuid: "e1aa938182db4af6a4deb259a5281368"
format_stamp: "Formatted at 2026-06-19 14:36:37 on dist-test-slave-70rm"
server_key: "e297ca62cc41f0531b1c31da48631397"
server_key_iv: "c36b63aaa19880c925da637563ad795e"
server_key_version: "encryptionkey@0"
I20260619 14:36:37.103297 24202 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260619 14:36:37.105664 24217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:37.107084 24202 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:36:37.107234 24202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781879514993456-2056-0/minicluster-data/master-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781879514993456-2056-0/minicluster-data/master-0/wal
uuid: "e1aa938182db4af6a4deb259a5281368"
format_stamp: "Formatted at 2026-06-19 14:36:37 on dist-test-slave-70rm"
server_key: "e297ca62cc41f0531b1c31da48631397"
server_key_iv: "c36b63aaa19880c925da637563ad795e"
server_key_version: "encryptionkey@0"
I20260619 14:36:37.107359 24202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781879514993456-2056-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781879514993456-2056-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781879514993456-2056-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:36:37.112138 24202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:36:37.115129 24202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:36:37.115334 24202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:36:37.122798 24202 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.62:43147
I20260619 14:36:37.122817 24269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.62:43147 every 8 connection(s)
I20260619 14:36:37.123797 24202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
I20260619 14:36:37.124961 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 24202
I20260619 14:36:37.125093 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781879514993456-2056-0/minicluster-data/master-0/wal/instance
I20260619 14:36:37.125407 2056 external_mini_cluster.cc:1428] Setting key c8bde048e66bda7931361bf0624939bd
I20260619 14:36:37.127133 24270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:36:37.130674 24273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:37.126814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37183 (local address 127.2.2.62:43147)
0619 14:36:37.127208 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:37.127228 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:37.127273 (+ 45us) server_negotiation.cc:407] Connection header received
0619 14:36:37.127909 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:37.127925 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:37.128221 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:37.128510 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:37.129341 (+ 831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:37.129366 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:37.129379 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:36:37.129391 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:37.129486 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:36:37.129507 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:37.129618 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:37.129720 (+ 102us) server_negotiation.cc:299] Negotiation successful
0619 14:36:37.129887 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":139,"threads_started":1}
I20260619 14:36:37.133517 24270 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1aa938182db4af6a4deb259a5281368: Bootstrap starting.
I20260619 14:36:37.134871 2056 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:36:37.136195 24270 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1aa938182db4af6a4deb259a5281368: Neither blocks nor log segments found. Creating new log.
I20260619 14:36:37.137070 24270 log.cc:824] T 00000000000000000000000000000000 P e1aa938182db4af6a4deb259a5281368: Log is configured to *not* fsync() on all Append() calls
I20260619 14:36:37.138872 24273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:37.137098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45815 (local address 127.2.2.62:43147)
0619 14:36:37.137234 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:37.137241 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:37.137359 (+ 118us) server_negotiation.cc:407] Connection header received
0619 14:36:37.137445 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:37.137450 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:37.137490 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:37.137575 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:37.138503 (+ 928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:37.138506 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:37.138509 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:36:37.138520 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:37.138547 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:36:37.138559 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:37.138645 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:37.138722 (+ 77us) server_negotiation.cc:299] Negotiation successful
0619 14:36:37.138763 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 14:36:37.139529 24270 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1aa938182db4af6a4deb259a5281368: No bootstrap required, opened a new log
I20260619 14:36:37.142926 24270 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1aa938182db4af6a4deb259a5281368 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1aa938182db4af6a4deb259a5281368" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 43147 } }
I20260619 14:36:37.143138 24270 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1aa938182db4af6a4deb259a5281368 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:36:37.143208 24270 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1aa938182db4af6a4deb259a5281368 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1aa938182db4af6a4deb259a5281368, State: Initialized, Role: FOLLOWER
I20260619 14:36:37.143676 24270 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1aa938182db4af6a4deb259a5281368 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1aa938182db4af6a4deb259a5281368" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 43147 } }
I20260619 14:36:37.143848 24270 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1aa938182db4af6a4deb259a5281368 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:36:37.143914 24270 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1aa938182db4af6a4deb259a5281368 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:36:37.144008 24270 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1aa938182db4af6a4deb259a5281368 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:36:37.144879 24270 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1aa938182db4af6a4deb259a5281368 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1aa938182db4af6a4deb259a5281368" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 43147 } }
I20260619 14:36:37.145208 24270 leader_election.cc:302] T 00000000000000000000000000000000 P e1aa938182db4af6a4deb259a5281368 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1aa938182db4af6a4deb259a5281368; no voters:
I20260619 14:36:37.145509 24270 leader_election.cc:288] T 00000000000000000000000000000000 P e1aa938182db4af6a4deb259a5281368 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:36:37.145635 24280 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1aa938182db4af6a4deb259a5281368 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:36:37.145875 24280 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1aa938182db4af6a4deb259a5281368 [term 1 LEADER]: Becoming Leader. State: Replica: e1aa938182db4af6a4deb259a5281368, State: Running, Role: LEADER
I20260619 14:36:37.146301 24280 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1aa938182db4af6a4deb259a5281368 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1aa938182db4af6a4deb259a5281368" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 43147 } }
I20260619 14:36:37.146827 24270 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1aa938182db4af6a4deb259a5281368 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:36:37.148015 24282 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1aa938182db4af6a4deb259a5281368 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1aa938182db4af6a4deb259a5281368. Latest consensus state: current_term: 1 leader_uuid: "e1aa938182db4af6a4deb259a5281368" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1aa938182db4af6a4deb259a5281368" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 43147 } } }
I20260619 14:36:37.148138 24282 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1aa938182db4af6a4deb259a5281368 [sys.catalog]: This master's current role is: LEADER
I20260619 14:36:37.148519 24289 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:36:37.148481 24281 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1aa938182db4af6a4deb259a5281368 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1aa938182db4af6a4deb259a5281368" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1aa938182db4af6a4deb259a5281368" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 43147 } } }
I20260619 14:36:37.148675 24281 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1aa938182db4af6a4deb259a5281368 [sys.catalog]: This master's current role is: LEADER
I20260619 14:36:37.151582 24289 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:36:37.157208 24289 catalog_manager.cc:1347] Generated new cluster ID: 4eaa39fbd86f482f97e2a7806900f841
I20260619 14:36:37.157289 24289 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:36:37.183063 24289 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:36:37.183432 24289 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:36:37.190629 24289 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e1aa938182db4af6a4deb259a5281368: Generated new TSK 0
I20260619 14:36:37.191376 24289 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:36:37.205842 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 24202
2026-06-19T14:36:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1394 ms)
[----------] 18 tests from AuthTokenIssuingTest (25434 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:36:37 dist-test-slave-70rm krb5kdc[24301](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:36:37 dist-test-slave-70rm krb5kdc[24301](info): set up 2 sockets
Jun 19 14:36:37 dist-test-slave-70rm krb5kdc[24301](info): commencing operation
krb5kdc: starting...
W20260619 14:36:38.234501 2056 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Jun 19 14:36:38 dist-test-slave-70rm krb5kdc[24301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:36:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:36:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.2.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab.
Entry for principal kudu/127.2.2.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab.
I20260619 14:36:38.357681 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:41993
--webserver_interface=127.2.2.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:33451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.2.62:41993,127.2.2.61:43219,127.2.2.60:45105
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:36:38.472811 24314 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 14:36:38.473212 24314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:36:38.473287 24314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:36:38.473376 24314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:36:38.477519 24314 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:36:38.477636 24314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:36:38.477680 24314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:36:38.477723 24314 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:36:38.477762 24314 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:36:38.483310 24314 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:33451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.2.62:41993,127.2.2.61:43219,127.2.2.60:45105
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:41993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.2.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24314
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:36:38.484540 24314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:36:38.485661 24314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:36:38.494694 24319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:38.494694 24322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:38.495167 24320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:38.495229 24314 server_base.cc:1034] running on GCE node
I20260619 14:36:38.495891 24314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:36:38.497083 24314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:36:38.498219 24314 hybrid_clock.cc:648] HybridClock initialized: now 1781879798498187 us; error 49 us; skew 500 ppm
Jun 19 14:36:38 dist-test-slave-70rm krb5kdc[24301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879798, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:36:38.500911 24314 init.cc:377] Logged in from keytab as kudu/127.2.2.62@KRBTEST.COM (short username kudu)
I20260619 14:36:38.501847 24314 webserver.cc:460] Webserver started at http://127.2.2.62:32977/ using document root <none> and password file <none>
I20260619 14:36:38.502487 24314 fs_manager.cc:359] Metadata directory not provided
I20260619 14:36:38.502552 24314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:36:38.502785 24314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:36:38.504921 24314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/minicluster-data/master-0/data/instance:
uuid: "0a10accd949c45869b4e4fd49f0ea5fc"
format_stamp: "Formatted at 2026-06-19 14:36:38 on dist-test-slave-70rm"
server_key: "c56d2b627a1dece5111514cd83cbe2bc"
server_key_iv: "15b24dc692da0ed645a5fa3de4ecfc0c"
server_key_version: "encryptionkey@0"
I20260619 14:36:38.505510 24314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/minicluster-data/master-0/wal/instance:
uuid: "0a10accd949c45869b4e4fd49f0ea5fc"
format_stamp: "Formatted at 2026-06-19 14:36:38 on dist-test-slave-70rm"
server_key: "c56d2b627a1dece5111514cd83cbe2bc"
server_key_iv: "15b24dc692da0ed645a5fa3de4ecfc0c"
server_key_version: "encryptionkey@0"
I20260619 14:36:38.509416 24314 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 14:36:38.511930 24329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:38.513226 24314 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 14:36:38.513371 24314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/minicluster-data/master-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/minicluster-data/master-0/wal
uuid: "0a10accd949c45869b4e4fd49f0ea5fc"
format_stamp: "Formatted at 2026-06-19 14:36:38 on dist-test-slave-70rm"
server_key: "c56d2b627a1dece5111514cd83cbe2bc"
server_key_iv: "15b24dc692da0ed645a5fa3de4ecfc0c"
server_key_version: "encryptionkey@0"
I20260619 14:36:38.513481 24314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:36:38.532721 24314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:36:38.536377 24314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:36:38.536588 24314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:36:38.543916 24314 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.62:41993
I20260619 14:36:38.544219 24381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.62:41993 every 8 connection(s)
I20260619 14:36:38.545011 24314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
I20260619 14:36:38.547960 24382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:36:38.551913 24382 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41993 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:36:38.554512 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 24314
I20260619 14:36:38.554662 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/minicluster-data/master-0/wal/instance
I20260619 14:36:38.554960 2056 external_mini_cluster.cc:1428] Setting key ef4701485037c6cf3b3f3ee7a9e1c896
WARNING: no policy specified for kudu/127.2.2.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.61@KRBTEST.COM" created.
Jun 19 14:36:38 dist-test-slave-70rm krb5kdc[24301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879798, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.62@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:36:38.572311 24384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:38.554758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41993 (local address 127.0.0.1:53738)
0619 14:36:38.556370 (+ 1612us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:38.556430 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 14:36:38.557428 (+ 998us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:36:38.558266 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:36:38.558274 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:36:38.558820 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:36:38.559257 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:38.559283 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:38.560548 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:38.560553 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:38.561909 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:38.561923 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:38.562661 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:38.562735 (+ 74us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:38.562861 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:38.563538 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:36:38.563567 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:36:38.566914 (+ 3347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:36:38.569093 (+ 2179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:36:38.569102 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:36:38.569112 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:36:38.569382 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:36:38.569814 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:36:38.569816 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:36:38.569818 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:36:38.569929 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:36:38.570836 (+ 907us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:36:38.570841 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:36:38.570978 (+ 137us) client_negotiation.cc:769] Sending connection context
0619 14:36:38.571125 (+ 147us) client_negotiation.cc:240] Negotiation successful
0619 14:36:38.571444 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1384,"thread_start_us":186,"threads_started":1}
I20260619 14:36:38.572317 24385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:38.555840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53738 (local address 127.2.2.62:41993)
0619 14:36:38.556202 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:38.556217 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:38.556576 (+ 359us) server_negotiation.cc:407] Connection header received
0619 14:36:38.557790 (+ 1214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:38.557796 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:38.558010 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:38.558236 (+ 226us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:38.559458 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:38.560389 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:38.562063 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:38.562552 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:38.562685 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:38.567077 (+ 4392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:38.567084 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:38.567096 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:36:38.567120 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:38.568974 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:38.569534 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:38.569544 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:38.569549 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:38.569715 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:38.570035 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:38.570041 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:38.570046 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:38.570705 (+ 659us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:38.570882 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:38.571292 (+ 410us) server_negotiation.cc:299] Negotiation successful
0619 14:36:38.571568 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":210,"thread_start_us":115,"threads_started":1}
I20260619 14:36:38.575109 24382 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.2.61" port: 43219 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:36:38.575816 24384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:38.575559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.61:43219 (local address 127.0.0.1:36174)
0619 14:36:38.575685 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:38.575756 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.61:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 14:36:38.576037 24331 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.2.61:43219: connect: Connection refused (error 111)
I20260619 14:36:38.576997 24384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:38.576752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.61:43219 (local address 127.0.0.1:36190)
0619 14:36:38.576891 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:38.576940 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.61:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260619 14:36:38.577153 24382 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.2.61:43219: Network error: Client connection negotiation failed: client connection to 127.2.2.61:43219: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.61.keytab.
Entry for principal kudu/127.2.2.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.61.keytab.
I20260619 14:36:38.582686 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.61:43219
--webserver_interface=127.2.2.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.61.keytab
--principal=kudu/127.2.2.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:33451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.2.62:41993,127.2.2.61:43219,127.2.2.60:45105
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:36:38.626422 24382 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.2.61" port: 43219 } attempt: 1
I20260619 14:36:38.627398 24384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:38.627083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.61:43219 (local address 127.0.0.1:36202)
0619 14:36:38.627253 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:38.627321 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.61:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260619 14:36:38.628170 24384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:38.627934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.61:43219 (local address 127.0.0.1:36206)
0619 14:36:38.628070 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:38.628115 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.61:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 14:36:38.628299 24382 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.2.61:43219: Network error: Client connection negotiation failed: client connection to 127.2.2.61:43219: connect: Connection refused (error 111)
W20260619 14:36:38.695228 24389 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 14:36:38.695586 24389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:36:38.695640 24389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:36:38.695714 24389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 14:36:38.696461 24382 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.2.61" port: 43219 } attempt: 2
I20260619 14:36:38.697278 24384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:38.696987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.61:43219 (local address 127.0.0.1:36208)
0619 14:36:38.697128 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:38.697195 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.61:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 14:36:38.697974 24384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:38.697783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.61:43219 (local address 127.0.0.1:36212)
0619 14:36:38.697868 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:38.697919 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.61:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260619 14:36:38.698087 24382 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.2.61:43219: Network error: Client connection negotiation failed: client connection to 127.2.2.61:43219: connect: Connection refused (error 111)
W20260619 14:36:38.700079 24389 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:36:38.700156 24389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:36:38.700202 24389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:36:38.700254 24389 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:36:38.700294 24389 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:36:38.705740 24389 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:33451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.2.62:41993,127.2.2.61:43219,127.2.2.60:45105
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.61.keytab
--principal=kudu/127.2.2.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.61:43219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.2.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:36:38.706930 24389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:36:38.707964 24389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:36:38.716431 24394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:38.716569 24395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:38.716408 24397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:38.716811 24389 server_base.cc:1034] running on GCE node
I20260619 14:36:38.717520 24389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:36:38.718845 24389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:36:38.720042 24389 hybrid_clock.cc:648] HybridClock initialized: now 1781879798720025 us; error 28 us; skew 500 ppm
Jun 19 14:36:38 dist-test-slave-70rm krb5kdc[24301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879798, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:36:38.723217 24389 init.cc:377] Logged in from keytab as kudu/127.2.2.61@KRBTEST.COM (short username kudu)
I20260619 14:36:38.724278 24389 webserver.cc:460] Webserver started at http://127.2.2.61:45447/ using document root <none> and password file <none>
I20260619 14:36:38.724951 24389 fs_manager.cc:359] Metadata directory not provided
I20260619 14:36:38.725025 24389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:36:38.725281 24389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:36:38.727717 24389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/minicluster-data/master-1/data/instance:
uuid: "958e5a1371894edda1b2142bbb85a482"
format_stamp: "Formatted at 2026-06-19 14:36:38 on dist-test-slave-70rm"
server_key: "9ab17426c5f68dbed6f705334271a50a"
server_key_iv: "b37f2f1a60f01792c8e0c6b278301af9"
server_key_version: "encryptionkey@0"
I20260619 14:36:38.728468 24389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/minicluster-data/master-1/wal/instance:
uuid: "958e5a1371894edda1b2142bbb85a482"
format_stamp: "Formatted at 2026-06-19 14:36:38 on dist-test-slave-70rm"
server_key: "9ab17426c5f68dbed6f705334271a50a"
server_key_iv: "b37f2f1a60f01792c8e0c6b278301af9"
server_key_version: "encryptionkey@0"
I20260619 14:36:38.732764 24389 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 14:36:38.735333 24404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:38.736668 24389 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:36:38.736802 24389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/minicluster-data/master-1/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/minicluster-data/master-1/wal
uuid: "958e5a1371894edda1b2142bbb85a482"
format_stamp: "Formatted at 2026-06-19 14:36:38 on dist-test-slave-70rm"
server_key: "9ab17426c5f68dbed6f705334271a50a"
server_key_iv: "b37f2f1a60f01792c8e0c6b278301af9"
server_key_version: "encryptionkey@0"
I20260619 14:36:38.736915 24389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:36:38.748903 24389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:36:38.751894 24389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:36:38.752102 24389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:36:38.758885 24389 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.61:43219
I20260619 14:36:38.759182 24456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.61:43219 every 8 connection(s)
I20260619 14:36:38.759878 24389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/minicluster-data/master-1/data/info.pb
I20260619 14:36:38.762656 24457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:36:38.766422 24457 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41993 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:36:38.768965 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 24389
I20260619 14:36:38.769110 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/minicluster-data/master-1/wal/instance
I20260619 14:36:38.769397 2056 external_mini_cluster.cc:1428] Setting key b09b5e0cefdca794fcdd2f19685b8f20
Jun 19 14:36:38 dist-test-slave-70rm krb5kdc[24301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879798, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.61@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.2.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.60@KRBTEST.COM" created.
I20260619 14:36:38.780776 24385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:38.768406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53740 (local address 127.2.2.62:41993)
0619 14:36:38.768554 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:38.768558 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:38.769509 (+ 951us) server_negotiation.cc:407] Connection header received
0619 14:36:38.771024 (+ 1515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:38.771029 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:38.771099 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:38.771208 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:38.772367 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:38.772900 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:38.774111 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:38.774395 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:38.774508 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:38.776865 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:38.776869 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:38.776871 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:36:38.776901 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:38.778326 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:38.778886 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:38.778891 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:38.778893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:38.778971 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:38.779354 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:38.779357 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:38.779358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:38.779735 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:38.779856 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:38.780371 (+ 515us) server_negotiation.cc:299] Negotiation successful
0619 14:36:38.780518 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 14:36:38.781414 24459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:38.768781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41993 (local address 127.0.0.1:53740)
0619 14:36:38.769325 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:38.769384 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 14:36:38.770730 (+ 1346us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:36:38.771378 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:36:38.771386 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:36:38.771727 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:36:38.772214 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:38.772227 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:38.773026 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:38.773030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:38.773984 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:38.773994 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:38.774492 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:38.774552 (+ 60us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:38.774729 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:38.775300 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:36:38.775316 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:36:38.776693 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:36:38.778436 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:36:38.778443 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:36:38.778458 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:36:38.778785 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:36:38.779076 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:36:38.779079 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:36:38.779081 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:36:38.779231 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:36:38.779845 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:36:38.779850 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:36:38.780098 (+ 248us) client_negotiation.cc:769] Sending connection context
0619 14:36:38.780266 (+ 168us) client_negotiation.cc:240] Negotiation successful
0619 14:36:38.780523 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":127,"threads_started":1}
I20260619 14:36:38.782860 24457 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.2.61" port: 43219 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:36:38.789268 24382 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.2.61" port: 43219 } attempt: 3
Jun 19 14:36:38 dist-test-slave-70rm krb5kdc[24301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879798, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.61@KRBTEST.COM for kudu/127.2.2.61@KRBTEST.COM
Jun 19 14:36:38 dist-test-slave-70rm krb5kdc[24301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879798, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.62@KRBTEST.COM for kudu/127.2.2.61@KRBTEST.COM
I20260619 14:36:38.795894 24459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:38.783414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.61:43219 (local address 127.0.0.1:36220)
0619 14:36:38.783523 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:38.783539 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:36:38.783618 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:36:38.784310 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:36:38.784313 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:36:38.784455 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:36:38.784541 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:38.784545 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:38.786059 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:38.786062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:38.786702 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:38.786710 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:38.787581 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:38.787598 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:38.787673 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:38.788197 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:36:38.788209 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:36:38.791329 (+ 3120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:36:38.793874 (+ 2545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:36:38.793878 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:36:38.793880 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:36:38.794226 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:36:38.794604 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:36:38.794607 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:36:38.794609 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:36:38.794732 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:36:38.795245 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:36:38.795249 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:36:38.795384 (+ 135us) client_negotiation.cc:769] Sending connection context
0619 14:36:38.795504 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 14:36:38.795648 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260619 14:36:38.796228 24462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:38.783491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36220 (local address 127.2.2.61:43219)
0619 14:36:38.783849 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:38.783863 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:38.783890 (+ 27us) server_negotiation.cc:407] Connection header received
0619 14:36:38.783935 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:38.783941 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:38.784158 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:38.784309 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:38.784648 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:38.785933 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:38.786819 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:38.787458 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:38.787592 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:38.791504 (+ 3912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:38.791514 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:38.791528 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:36:38.791549 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:38.793761 (+ 2212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:38.794358 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:38.794368 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:38.794373 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:38.794482 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:38.794830 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:38.794833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:38.794834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:38.795130 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:38.795318 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:38.795593 (+ 275us) server_negotiation.cc:299] Negotiation successful
0619 14:36:38.795745 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":167,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.60.keytab.
Entry for principal kudu/127.2.2.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.60.keytab.
I20260619 14:36:38.797497 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.60:45105
--webserver_interface=127.2.2.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.60.keytab
--principal=kudu/127.2.2.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:33451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.2.62:41993,127.2.2.61:43219,127.2.2.60:45105
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:36:38.798630 24457 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.2.60" port: 45105 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:36:38.799479 24459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:38.799165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.60:45105 (local address 127.0.0.1:54520)
0619 14:36:38.799306 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:38.799387 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.60:45105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260619 14:36:38.799690 24407 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.2.60:45105: connect: Connection refused (error 111)
I20260619 14:36:38.800129 24463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:38.789926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36226 (local address 127.2.2.61:43219)
0619 14:36:38.790389 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:38.790399 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:38.790416 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:36:38.790460 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:38.790466 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:38.790521 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:38.790630 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:38.791161 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:38.791902 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:38.793285 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:38.793669 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:38.793794 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:38.796197 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:38.796201 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:38.796204 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:36:38.796228 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:38.798085 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:38.798722 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:38.798728 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:38.798732 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:38.798822 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:38.799158 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:38.799160 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:38.799162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:38.799400 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:38.799503 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:38.799853 (+ 350us) server_negotiation.cc:299] Negotiation successful
0619 14:36:38.799983 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":140,"threads_started":1}
I20260619 14:36:38.800163 24384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:38.789865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.61:43219 (local address 127.0.0.1:36226)
0619 14:36:38.790062 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:38.790082 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 14:36:38.790252 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:36:38.790678 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:36:38.790681 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:36:38.790862 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:36:38.791018 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:38.791026 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:38.792025 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:38.792030 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:38.793169 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:38.793180 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:38.793797 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:38.793850 (+ 53us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:38.794001 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:38.794601 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:36:38.794641 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:36:38.796001 (+ 1360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:36:38.798273 (+ 2272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:36:38.798279 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:36:38.798285 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:36:38.798573 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:36:38.798941 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:36:38.798946 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:36:38.798951 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:36:38.799054 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:36:38.799516 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:36:38.799521 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:36:38.799653 (+ 132us) client_negotiation.cc:769] Sending connection context
0619 14:36:38.799782 (+ 129us) client_negotiation.cc:240] Negotiation successful
0619 14:36:38.799962 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 14:36:38.800887 24382 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.2.60" port: 45105 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:36:38.801060 24459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:38.800804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.60:45105 (local address 127.0.0.1:54524)
0619 14:36:38.800937 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:38.800975 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.60:45105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260619 14:36:38.801208 24457 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.2.60:45105: Network error: Client connection negotiation failed: client connection to 127.2.2.60:45105: connect: Connection refused (error 111)
I20260619 14:36:38.801549 24384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:38.801334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.60:45105 (local address 127.0.0.1:54530)
0619 14:36:38.801444 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:38.801494 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.60:45105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260619 14:36:38.802268 24384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:38.801971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.60:45105 (local address 127.0.0.1:54538)
0619 14:36:38.802169 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:38.802216 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.60:45105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260619 14:36:38.802443 24382 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.2.60:45105: Network error: Client connection negotiation failed: client connection to 127.2.2.60:45105: connect: Connection refused (error 111)
I20260619 14:36:38.833600 24382 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.2.60" port: 45105 } attempt: 1
I20260619 14:36:38.834528 24384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:38.834230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.60:45105 (local address 127.0.0.1:54550)
0619 14:36:38.834386 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:38.834451 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.60:45105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 14:36:38.835189 24384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:38.835014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.60:45105 (local address 127.0.0.1:54556)
0619 14:36:38.835097 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:38.835139 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.60:45105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260619 14:36:38.835319 24382 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.2.60:45105: Network error: Client connection negotiation failed: client connection to 127.2.2.60:45105: connect: Connection refused (error 111)
I20260619 14:36:38.850485 24457 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.2.60" port: 45105 } attempt: 1
I20260619 14:36:38.851409 24459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:38.851105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.60:45105 (local address 127.0.0.1:54572)
0619 14:36:38.851265 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:38.851332 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.60:45105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260619 14:36:38.852138 24459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:38.851924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.60:45105 (local address 127.0.0.1:54582)
0619 14:36:38.852029 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:38.852081 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.60:45105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 14:36:38.852268 24457 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.2.60:45105: Network error: Client connection negotiation failed: client connection to 127.2.2.60:45105: connect: Connection refused (error 111)
I20260619 14:36:38.910493 24382 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.2.60" port: 45105 } attempt: 2
I20260619 14:36:38.911448 24384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:38.911071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.60:45105 (local address 127.0.0.1:54596)
0619 14:36:38.911249 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:38.911349 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.60:45105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260619 14:36:38.911212 24464 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 14:36:38.911648 24464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:36:38.911697 24464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:36:38.911770 24464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 14:36:38.912395 24384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:38.912156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.60:45105 (local address 127.0.0.1:54600)
0619 14:36:38.912280 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:38.912343 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.60:45105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260619 14:36:38.912989 24382 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.2.60:45105: Network error: Client connection negotiation failed: client connection to 127.2.2.60:45105: connect: Connection refused (error 111)
W20260619 14:36:38.916424 24464 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:36:38.916479 24464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:36:38.916499 24464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:36:38.916535 24464 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:36:38.916574 24464 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:36:38.920424 24457 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.2.60" port: 45105 } attempt: 2
I20260619 14:36:38.921103 24459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:38.920852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.60:45105 (local address 127.0.0.1:54606)
0619 14:36:38.920972 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:38.921038 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.60:45105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 14:36:38.921761 24459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:38.921564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.60:45105 (local address 127.0.0.1:54614)
0619 14:36:38.921661 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:38.921711 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.60:45105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260619 14:36:38.921939 24457 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.2.60:45105: Network error: Client connection negotiation failed: client connection to 127.2.2.60:45105: connect: Connection refused (error 111)
I20260619 14:36:38.922163 24464 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:33451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.2.62:41993,127.2.2.61:43219,127.2.2.60:45105
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.60.keytab
--principal=kudu/127.2.2.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.60:45105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.2.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:36:38.923202 24464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:36:38.924216 24464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:36:38.933301 24464 server_base.cc:1034] running on GCE node
W20260619 14:36:38.933301 24471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:38.933303 24470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:38.933499 24473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:38.934034 24464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:36:38.935274 24464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:36:38.936437 24464 hybrid_clock.cc:648] HybridClock initialized: now 1781879798936403 us; error 50 us; skew 500 ppm
Jun 19 14:36:38 dist-test-slave-70rm krb5kdc[24301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879798, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:36:38.939270 24464 init.cc:377] Logged in from keytab as kudu/127.2.2.60@KRBTEST.COM (short username kudu)
I20260619 14:36:38.940249 24464 webserver.cc:460] Webserver started at http://127.2.2.60:41807/ using document root <none> and password file <none>
I20260619 14:36:38.940793 24464 fs_manager.cc:359] Metadata directory not provided
I20260619 14:36:38.940874 24464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:36:38.941145 24464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:36:38.943198 24464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/minicluster-data/master-2/data/instance:
uuid: "7074277d8a324bb594b81b92043bb66a"
format_stamp: "Formatted at 2026-06-19 14:36:38 on dist-test-slave-70rm"
server_key: "a667b075363ff1800c4e669cb66aac56"
server_key_iv: "d3a0ccc273f2d6d0d2dbdf3531744006"
server_key_version: "encryptionkey@0"
I20260619 14:36:38.943732 24464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/minicluster-data/master-2/wal/instance:
uuid: "7074277d8a324bb594b81b92043bb66a"
format_stamp: "Formatted at 2026-06-19 14:36:38 on dist-test-slave-70rm"
server_key: "a667b075363ff1800c4e669cb66aac56"
server_key_iv: "d3a0ccc273f2d6d0d2dbdf3531744006"
server_key_version: "encryptionkey@0"
I20260619 14:36:38.947222 24464 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260619 14:36:38.949466 24480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:38.950644 24464 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:36:38.950770 24464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/minicluster-data/master-2/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/minicluster-data/master-2/wal
uuid: "7074277d8a324bb594b81b92043bb66a"
format_stamp: "Formatted at 2026-06-19 14:36:38 on dist-test-slave-70rm"
server_key: "a667b075363ff1800c4e669cb66aac56"
server_key_iv: "d3a0ccc273f2d6d0d2dbdf3531744006"
server_key_version: "encryptionkey@0"
I20260619 14:36:38.950870 24464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:36:38.992053 24464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:36:38.995460 24464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:36:38.995669 24464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:36:39.002946 24464 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.60:45105
I20260619 14:36:39.003250 24532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.60:45105 every 8 connection(s)
I20260619 14:36:39.003994 24464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/minicluster-data/master-2/data/info.pb
I20260619 14:36:39.005304 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 24464
I20260619 14:36:39.005446 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879514993456-2056-0/minicluster-data/master-2/wal/instance
I20260619 14:36:39.005765 2056 external_mini_cluster.cc:1428] Setting key 8c4d9a5f1c15dbaa26644cb69c40867c
I20260619 14:36:39.007918 24533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:36:39.012184 24382 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.2.60" port: 45105 } attempt: 3
I20260619 14:36:39.011869 24533 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41993 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:36:39.013119 24457 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.2.60" port: 45105 } attempt: 3
Jun 19 14:36:39 dist-test-slave-70rm krb5kdc[24301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.2.60@KRBTEST.COM
Jun 19 14:36:39 dist-test-slave-70rm krb5kdc[24301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879798, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.62@KRBTEST.COM for kudu/127.2.2.60@KRBTEST.COM
Jun 19 14:36:39 dist-test-slave-70rm krb5kdc[24301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879798, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.61@KRBTEST.COM for kudu/127.2.2.60@KRBTEST.COM
Jun 19 14:36:39 dist-test-slave-70rm krb5kdc[24301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879798, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.60@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:36:39.025424 24459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:39.013551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.60:45105 (local address 127.0.0.1:54650)
0619 14:36:39.013736 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:39.013749 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:36:39.013829 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:36:39.014381 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:36:39.014384 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:36:39.014594 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:36:39.014709 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:39.014716 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:39.015918 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:39.015923 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:39.016757 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:39.016768 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:39.017428 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:39.017446 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:39.017527 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:39.018050 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:36:39.018068 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:36:39.019746 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:36:39.022808 (+ 3062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:36:39.022815 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:36:39.022820 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:36:39.023098 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:36:39.024431 (+ 1333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:36:39.024434 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:36:39.024436 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:36:39.024550 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:36:39.024964 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:36:39.024967 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:36:39.025072 (+ 105us) client_negotiation.cc:769] Sending connection context
0619 14:36:39.025151 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 14:36:39.025276 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260619 14:36:39.025326 24384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:39.012691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.60:45105 (local address 127.0.0.1:54642)
0619 14:36:39.012818 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:39.012834 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:36:39.012925 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:36:39.013405 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:36:39.013411 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:36:39.013621 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:36:39.013738 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:39.013748 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:39.015162 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:39.015168 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:39.015966 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:39.015979 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:39.016550 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:39.016566 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:39.016652 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:39.017152 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:36:39.017167 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:36:39.019434 (+ 2267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:36:39.023193 (+ 3759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:36:39.023197 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:36:39.023199 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:36:39.023626 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:36:39.024006 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:36:39.024011 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:36:39.024016 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:36:39.024149 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:36:39.024651 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:36:39.024656 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:36:39.024775 (+ 119us) client_negotiation.cc:769] Sending connection context
0619 14:36:39.024936 (+ 161us) client_negotiation.cc:240] Negotiation successful
0619 14:36:39.025114 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 14:36:39.026225 24537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:39.012713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54642 (local address 127.2.2.60:45105)
0619 14:36:39.013064 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:39.013100 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:39.013122 (+ 22us) server_negotiation.cc:407] Connection header received
0619 14:36:39.013163 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:39.013166 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:39.013210 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:39.013849 (+ 639us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:39.013997 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:39.015070 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:39.016094 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:39.016427 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:39.017998 (+ 1571us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:39.019589 (+ 1591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:39.019595 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:39.019600 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:36:39.019632 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:39.023077 (+ 3445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:39.023740 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:39.023743 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:39.023745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:39.023831 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:39.024291 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:39.024297 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:39.024302 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:39.024510 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:39.025398 (+ 888us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:39.025473 (+ 75us) server_negotiation.cc:299] Negotiation successful
0619 14:36:39.025597 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":99,"threads_started":1}
I20260619 14:36:39.026206 24536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:39.007337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54628 (local address 127.2.2.60:45105)
0619 14:36:39.007794 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:39.007809 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:39.007846 (+ 37us) server_negotiation.cc:407] Connection header received
0619 14:36:39.008774 (+ 928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:39.008792 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:39.009142 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:39.009500 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:39.010378 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:39.011412 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:39.012893 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:39.014241 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:39.014392 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:39.018900 (+ 4508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:39.018905 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:39.018916 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:36:39.018980 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:39.022007 (+ 3027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:39.022698 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:39.022707 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:39.022718 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:39.022887 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:39.023267 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:39.023271 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:39.023273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:39.023760 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:39.023999 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:39.024710 (+ 711us) server_negotiation.cc:299] Negotiation successful
0619 14:36:39.025050 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":319,"thread_start_us":157,"threads_started":1}
I20260619 14:36:39.026849 24538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:39.013812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54650 (local address 127.2.2.60:45105)
0619 14:36:39.014142 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:39.014151 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:39.014163 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:36:39.014194 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:39.014196 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:39.014234 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:39.014814 (+ 580us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:39.014895 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:39.015814 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:39.016921 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:39.017306 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:39.017758 (+ 452us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:39.020108 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:39.020114 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:39.020120 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:36:39.020148 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:39.022678 (+ 2530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:39.024195 (+ 1517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:39.024199 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:39.024201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:39.024315 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:39.024661 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:39.024664 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:39.024665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:39.024851 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:39.025855 (+ 1004us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:39.025929 (+ 74us) server_negotiation.cc:299] Negotiation successful
0619 14:36:39.026283 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"spinlock_wait_cycles":4352,"thread_start_us":148,"threads_started":1}
I20260619 14:36:39.026997 24385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:39.014254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53756 (local address 127.2.2.62:41993)
0619 14:36:39.014378 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:39.014382 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:39.015070 (+ 688us) server_negotiation.cc:407] Connection header received
0619 14:36:39.015273 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:39.015277 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:39.015325 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:39.015405 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:39.016246 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:39.017053 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:39.018527 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:39.018880 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:39.018958 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:39.021830 (+ 2872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:39.021834 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:39.021837 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:36:39.021862 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:39.024072 (+ 2210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:39.024887 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:39.024891 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:39.024893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:39.024977 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:39.025304 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:39.025307 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:39.025309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:39.025594 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:39.025698 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:39.026677 (+ 979us) server_negotiation.cc:299] Negotiation successful
0619 14:36:39.026796 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59,"spinlock_wait_cycles":1920}
I20260619 14:36:39.026996 24539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:39.014036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:41993 (local address 127.0.0.1:53756)
0619 14:36:39.014920 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:39.014973 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 14:36:39.015122 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:36:39.015471 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:36:39.015477 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:36:39.015921 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:36:39.016115 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:39.016124 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:39.017248 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:39.017252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:39.018395 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:39.018406 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:39.019108 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:39.019125 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:39.019242 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:39.019746 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:36:39.019761 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:36:39.021701 (+ 1940us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:36:39.024181 (+ 2480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:36:39.024193 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:36:39.024202 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:36:39.024765 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:36:39.025103 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:36:39.025106 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:36:39.025108 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:36:39.025197 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:36:39.026249 (+ 1052us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:36:39.026259 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:36:39.026391 (+ 132us) client_negotiation.cc:769] Sending connection context
0619 14:36:39.026600 (+ 209us) client_negotiation.cc:240] Negotiation successful
0619 14:36:39.026776 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"spinlock_wait_cycles":6272,"thread_start_us":107,"threads_started":1}
I20260619 14:36:39.028389 24533 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.2.61" port: 43219 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:36:39.031050 24457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 958e5a1371894edda1b2142bbb85a482: Bootstrap starting.
I20260619 14:36:39.033392 24382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a10accd949c45869b4e4fd49f0ea5fc: Bootstrap starting.
I20260619 14:36:39.034101 24457 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 958e5a1371894edda1b2142bbb85a482: Neither blocks nor log segments found. Creating new log.
I20260619 14:36:39.035135 24457 log.cc:824] T 00000000000000000000000000000000 P 958e5a1371894edda1b2142bbb85a482: Log is configured to *not* fsync() on all Append() calls
Jun 19 14:36:39 dist-test-slave-70rm krb5kdc[24301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879798, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.60@KRBTEST.COM for kudu/127.2.2.61@KRBTEST.COM
I20260619 14:36:39.036402 24382 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a10accd949c45869b4e4fd49f0ea5fc: Neither blocks nor log segments found. Creating new log.
I20260619 14:36:39.037345 24382 log.cc:824] T 00000000000000000000000000000000 P 0a10accd949c45869b4e4fd49f0ea5fc: Log is configured to *not* fsync() on all Append() calls
I20260619 14:36:39.037974 24457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 958e5a1371894edda1b2142bbb85a482: No bootstrap required, opened a new log
Jun 19 14:36:39 dist-test-slave-70rm krb5kdc[24301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:36:39.040230 24382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a10accd949c45869b4e4fd49f0ea5fc: No bootstrap required, opened a new log
I20260619 14:36:39.042474 24457 raft_consensus.cc:348] T 00000000000000000000000000000000 P 958e5a1371894edda1b2142bbb85a482 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a10accd949c45869b4e4fd49f0ea5fc" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41993 } } peers { permanent_uuid: "958e5a1371894edda1b2142bbb85a482" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 43219 } } peers { permanent_uuid: "7074277d8a324bb594b81b92043bb66a" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 45105 } }
I20260619 14:36:39.042753 24457 raft_consensus.cc:729] T 00000000000000000000000000000000 P 958e5a1371894edda1b2142bbb85a482 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 958e5a1371894edda1b2142bbb85a482, State: Initialized, Role: FOLLOWER
I20260619 14:36:39.043045 24385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:39.030358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53758 (local address 127.2.2.62:41993)
0619 14:36:39.030584 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:39.030589 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:39.030607 (+ 18us) server_negotiation.cc:407] Connection header received
0619 14:36:39.030645 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:39.030648 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:39.030693 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:39.030779 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:39.031821 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:39.032545 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:39.033725 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:39.034542 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:39.034636 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:39.039128 (+ 4492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:39.039132 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:39.039135 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:36:39.039162 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:39.040916 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:39.041638 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:39.041641 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:39.041643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:39.041731 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:39.042055 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:39.042058 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:39.042060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:39.042324 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:39.042524 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:39.042746 (+ 222us) server_negotiation.cc:299] Negotiation successful
0619 14:36:39.042878 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136}
I20260619 14:36:39.043128 24457 consensus_queue.cc:260] T 00000000000000000000000000000000 P 958e5a1371894edda1b2142bbb85a482 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a10accd949c45869b4e4fd49f0ea5fc" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41993 } } peers { permanent_uuid: "958e5a1371894edda1b2142bbb85a482" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 43219 } } peers { permanent_uuid: "7074277d8a324bb594b81b92043bb66a" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 45105 } }
I20260619 14:36:39.044320 24542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 958e5a1371894edda1b2142bbb85a482 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a10accd949c45869b4e4fd49f0ea5fc" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41993 } } peers { permanent_uuid: "958e5a1371894edda1b2142bbb85a482" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 43219 } } peers { permanent_uuid: "7074277d8a324bb594b81b92043bb66a" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 45105 } } }
I20260619 14:36:39.044484 24542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 958e5a1371894edda1b2142bbb85a482 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:36:39.044600 24457 sys_catalog.cc:564] T 00000000000000000000000000000000 P 958e5a1371894edda1b2142bbb85a482 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:36:39.044507 24382 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a10accd949c45869b4e4fd49f0ea5fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a10accd949c45869b4e4fd49f0ea5fc" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41993 } } peers { permanent_uuid: "958e5a1371894edda1b2142bbb85a482" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 43219 } } peers { permanent_uuid: "7074277d8a324bb594b81b92043bb66a" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 45105 } }
I20260619 14:36:39.044818 24382 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a10accd949c45869b4e4fd49f0ea5fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a10accd949c45869b4e4fd49f0ea5fc, State: Initialized, Role: FOLLOWER
I20260619 14:36:39.045177 24382 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a10accd949c45869b4e4fd49f0ea5fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a10accd949c45869b4e4fd49f0ea5fc" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41993 } } peers { permanent_uuid: "958e5a1371894edda1b2142bbb85a482" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 43219 } } peers { permanent_uuid: "7074277d8a324bb594b81b92043bb66a" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 45105 } }
I20260619 14:36:39.045665 24543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a10accd949c45869b4e4fd49f0ea5fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a10accd949c45869b4e4fd49f0ea5fc" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41993 } } peers { permanent_uuid: "958e5a1371894edda1b2142bbb85a482" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 43219 } } peers { permanent_uuid: "7074277d8a324bb594b81b92043bb66a" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 45105 } } }
I20260619 14:36:39.045841 24543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a10accd949c45869b4e4fd49f0ea5fc [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:36:39.046568 24382 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a10accd949c45869b4e4fd49f0ea5fc [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:36:39.050334 24463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:39.029034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36238 (local address 127.2.2.61:43219)
0619 14:36:39.030476 (+ 1442us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:39.030479 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:39.030490 (+ 11us) server_negotiation.cc:407] Connection header received
0619 14:36:39.030520 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:39.030522 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:39.030557 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:39.030666 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:39.031106 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:39.031846 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:39.033039 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:39.033378 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:39.033462 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:39.036096 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:39.036101 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:39.036104 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:36:39.036133 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:39.037984 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:39.048595 (+ 10611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:39.048598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:39.048600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:39.048728 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:39.049128 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:39.049131 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:39.049133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:39.049404 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:39.049498 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:39.049931 (+ 433us) server_negotiation.cc:299] Negotiation successful
0619 14:36:39.050094 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1375}
I20260619 14:36:39.050889 24539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:39.028941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.61:43219 (local address 127.0.0.1:36238)
0619 14:36:39.029046 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:39.029064 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:36:39.029149 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:36:39.030670 (+ 1521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:36:39.030673 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:36:39.030853 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:36:39.030971 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:39.030978 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:39.031992 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:39.031996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:39.032891 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:39.032903 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:39.033633 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:39.033650 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:39.033745 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:39.034355 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:36:39.034369 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:36:39.035925 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:36:39.038239 (+ 2314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:36:39.038243 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:36:39.038246 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:36:39.038539 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:36:39.048871 (+ 10332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:36:39.048875 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:36:39.048878 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:36:39.049003 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:36:39.049588 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:36:39.049593 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:36:39.049719 (+ 126us) client_negotiation.cc:769] Sending connection context
0619 14:36:39.050542 (+ 823us) client_negotiation.cc:240] Negotiation successful
0619 14:36:39.050723 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
Jun 19 14:36:39 dist-test-slave-70rm krb5kdc[24301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.2.61@KRBTEST.COM
I20260619 14:36:39.051585 24533 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.2.60" port: 45105 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:36:39.048544 24389 master_runner.cc:186] Error getting master registration for 127.2.2.62:41993: OK, instance_id { permanent_uuid: "0a10accd949c45869b4e4fd49f0ea5fc" instance_seqno: 1781879798541866 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 14:36:39.052139 24389 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260619 14:36:39.056064 24566 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0a10accd949c45869b4e4fd49f0ea5fc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:36:39.056154 24566 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:36:39.057118 24462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:39.044487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36242 (local address 127.2.2.61:43219)
0619 14:36:39.044615 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:39.044622 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:39.044953 (+ 331us) server_negotiation.cc:407] Connection header received
0619 14:36:39.045199 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:39.045202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:39.045254 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:39.045343 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:39.046209 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:39.046954 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:39.048179 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:39.048585 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:39.048678 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:39.052633 (+ 3955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:39.052638 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:39.052642 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:36:39.052666 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:39.054805 (+ 2139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:39.055304 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:39.055308 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:39.055309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:39.055391 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:39.055707 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:39.055710 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:39.055712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:39.055911 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:39.056386 (+ 475us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:39.056778 (+ 392us) server_negotiation.cc:299] Negotiation successful
0619 14:36:39.056932 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260619 14:36:39.058444 24554 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 958e5a1371894edda1b2142bbb85a482: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:36:39.058521 24554 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Jun 19 14:36:39 dist-test-slave-70rm krb5kdc[24301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879798, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.60@KRBTEST.COM for kudu/127.2.2.60@KRBTEST.COM
I20260619 14:36:39.055104 24314 master_runner.cc:186] Error getting master registration for 127.2.2.60:45105: OK, instance_id { permanent_uuid: "7074277d8a324bb594b81b92043bb66a" instance_seqno: 1781879799000912 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 14:36:39.060027 24314 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 14:36:39.060487 2056 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:36:39.061848 24356 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "0a10accd949c45869b4e4fd49f0ea5fc"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53758
I20260619 14:36:39.062101 24356 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a10accd949c45869b4e4fd49f0ea5fc [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 14:36:39.062263 24356 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a10accd949c45869b4e4fd49f0ea5fc [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:36:39.063666 24356 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a10accd949c45869b4e4fd49f0ea5fc [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a10accd949c45869b4e4fd49f0ea5fc" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41993 } } peers { permanent_uuid: "958e5a1371894edda1b2142bbb85a482" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 43219 } } peers { permanent_uuid: "7074277d8a324bb594b81b92043bb66a" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 45105 } }
I20260619 14:36:39.063907 24537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:39.052284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54654 (local address 127.2.2.60:45105)
0619 14:36:39.052397 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:39.052401 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:39.052844 (+ 443us) server_negotiation.cc:407] Connection header received
0619 14:36:39.052994 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:39.052997 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:39.053032 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:39.053075 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:39.053597 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:39.054335 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:39.055304 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:39.055634 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:39.055698 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:39.060395 (+ 4697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:39.060400 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:39.060402 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:36:39.060427 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:39.062214 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:39.062681 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:39.062684 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:39.062686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:39.062759 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:39.063024 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:39.063027 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:39.063029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:39.063233 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:39.063300 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:39.063617 (+ 317us) server_negotiation.cc:299] Negotiation successful
0619 14:36:39.063742 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 14:36:39.064630 24539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:39.052096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.60:45105 (local address 127.0.0.1:54654)
0619 14:36:39.052772 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:39.052789 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:36:39.052900 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:36:39.053200 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:36:39.053202 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:36:39.053379 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:36:39.053486 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:39.053493 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:39.054443 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:39.054446 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:39.055194 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:39.055204 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:39.055816 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:39.055829 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:39.055902 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:39.058646 (+ 2744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:36:39.058659 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:36:39.060237 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:36:39.062329 (+ 2092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:36:39.062332 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:36:39.062335 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:36:39.062578 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:36:39.062852 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:36:39.062855 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:36:39.062857 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:36:39.062927 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:36:39.063356 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:36:39.063359 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:36:39.063434 (+ 75us) client_negotiation.cc:769] Sending connection context
0619 14:36:39.064173 (+ 739us) client_negotiation.cc:240] Negotiation successful
0619 14:36:39.064325 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604}
I20260619 14:36:39.064713 24356 leader_election.cc:288] T 00000000000000000000000000000000 P 0a10accd949c45869b4e4fd49f0ea5fc [CANDIDATE]: Term 1 election: Requested vote from peers 958e5a1371894edda1b2142bbb85a482 (127.2.2.61:43219), 7074277d8a324bb594b81b92043bb66a (127.2.2.60:45105)
I20260619 14:36:39.066438 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a10accd949c45869b4e4fd49f0ea5fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "958e5a1371894edda1b2142bbb85a482"
I20260619 14:36:39.066807 24431 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 958e5a1371894edda1b2142bbb85a482 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:36:39.068284 24431 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 958e5a1371894edda1b2142bbb85a482 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a10accd949c45869b4e4fd49f0ea5fc in term 1.
I20260619 14:36:39.068416 24533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7074277d8a324bb594b81b92043bb66a: Bootstrap starting.
I20260619 14:36:39.071362 24533 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7074277d8a324bb594b81b92043bb66a: Neither blocks nor log segments found. Creating new log.
I20260619 14:36:39.072299 24533 log.cc:824] T 00000000000000000000000000000000 P 7074277d8a324bb594b81b92043bb66a: Log is configured to *not* fsync() on all Append() calls
I20260619 14:36:39.073101 24331 leader_election.cc:302] T 00000000000000000000000000000000 P 0a10accd949c45869b4e4fd49f0ea5fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a10accd949c45869b4e4fd49f0ea5fc, 958e5a1371894edda1b2142bbb85a482; no voters:
I20260619 14:36:39.073484 24543 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a10accd949c45869b4e4fd49f0ea5fc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:36:39.073716 24543 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a10accd949c45869b4e4fd49f0ea5fc [term 1 LEADER]: Becoming Leader. State: Replica: 0a10accd949c45869b4e4fd49f0ea5fc, State: Running, Role: LEADER
I20260619 14:36:39.074076 24543 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a10accd949c45869b4e4fd49f0ea5fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a10accd949c45869b4e4fd49f0ea5fc" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41993 } } peers { permanent_uuid: "958e5a1371894edda1b2142bbb85a482" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 43219 } } peers { permanent_uuid: "7074277d8a324bb594b81b92043bb66a" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 45105 } }
I20260619 14:36:39.076370 24543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a10accd949c45869b4e4fd49f0ea5fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a10accd949c45869b4e4fd49f0ea5fc. Latest consensus state: current_term: 1 leader_uuid: "0a10accd949c45869b4e4fd49f0ea5fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a10accd949c45869b4e4fd49f0ea5fc" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41993 } } peers { permanent_uuid: "958e5a1371894edda1b2142bbb85a482" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 43219 } } peers { permanent_uuid: "7074277d8a324bb594b81b92043bb66a" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 45105 } } }
I20260619 14:36:39.076534 24543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a10accd949c45869b4e4fd49f0ea5fc [sys.catalog]: This master's current role is: LEADER
I20260619 14:36:39.076964 24577 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:36:39.078094 24577 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:36:39.082437 24385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:39.070649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53772 (local address 127.2.2.62:41993)
0619 14:36:39.070765 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:39.070769 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:39.071962 (+ 1193us) server_negotiation.cc:407] Connection header received
0619 14:36:39.072184 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:39.072188 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:39.072237 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:39.072287 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:39.073676 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:39.074517 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:39.075762 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:39.076083 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:39.076172 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:39.078325 (+ 2153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:39.078329 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:39.078332 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:36:39.078358 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:39.080256 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:39.080809 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:39.080813 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:39.080815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:39.080904 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:39.081395 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:39.081401 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:39.081403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:39.081651 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:39.081755 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:39.082146 (+ 391us) server_negotiation.cc:299] Negotiation successful
0619 14:36:39.082278 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 14:36:39.084525 24533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7074277d8a324bb594b81b92043bb66a: No bootstrap required, opened a new log
I20260619 14:36:39.091271 24533 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7074277d8a324bb594b81b92043bb66a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a10accd949c45869b4e4fd49f0ea5fc" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41993 } } peers { permanent_uuid: "958e5a1371894edda1b2142bbb85a482" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 43219 } } peers { permanent_uuid: "7074277d8a324bb594b81b92043bb66a" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 45105 } }
I20260619 14:36:39.091588 24533 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7074277d8a324bb594b81b92043bb66a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7074277d8a324bb594b81b92043bb66a, State: Initialized, Role: FOLLOWER
I20260619 14:36:39.091969 24533 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7074277d8a324bb594b81b92043bb66a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a10accd949c45869b4e4fd49f0ea5fc" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41993 } } peers { permanent_uuid: "958e5a1371894edda1b2142bbb85a482" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 43219 } } peers { permanent_uuid: "7074277d8a324bb594b81b92043bb66a" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 45105 } }
I20260619 14:36:39.092754 24537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:39.071296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54666 (local address 127.2.2.60:45105)
0619 14:36:39.071413 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:39.071417 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:39.083335 (+ 11918us) server_negotiation.cc:407] Connection header received
0619 14:36:39.083531 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:39.083534 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:39.083580 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:39.083632 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:39.084466 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:39.085183 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:39.086352 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:39.086657 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:39.086736 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:39.088638 (+ 1902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:39.088643 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:39.088646 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:36:39.088673 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:39.090603 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:39.091125 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:39.091129 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:39.091130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:39.091215 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:39.091514 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:39.091517 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:39.091519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:39.091762 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:39.091884 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:39.092473 (+ 589us) server_negotiation.cc:299] Negotiation successful
0619 14:36:39.092610 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 14:36:39.097493 24533 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7074277d8a324bb594b81b92043bb66a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:36:39.097570 24507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a10accd949c45869b4e4fd49f0ea5fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7074277d8a324bb594b81b92043bb66a"
I20260619 14:36:39.097853 24507 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7074277d8a324bb594b81b92043bb66a [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:36:39.099298 24507 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7074277d8a324bb594b81b92043bb66a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a10accd949c45869b4e4fd49f0ea5fc in term 1.
I20260619 14:36:39.101389 24580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7074277d8a324bb594b81b92043bb66a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a10accd949c45869b4e4fd49f0ea5fc" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41993 } } peers { permanent_uuid: "958e5a1371894edda1b2142bbb85a482" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 43219 } } peers { permanent_uuid: "7074277d8a324bb594b81b92043bb66a" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 45105 } } }
I20260619 14:36:39.101564 24580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7074277d8a324bb594b81b92043bb66a [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:36:39.103118 24462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:39.082590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36254 (local address 127.2.2.61:43219)
0619 14:36:39.082730 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:39.082738 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:39.093641 (+ 10903us) server_negotiation.cc:407] Connection header received
0619 14:36:39.093834 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:39.093837 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:39.093893 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:39.093954 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:39.094865 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:39.095597 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:39.096614 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:39.096907 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:39.096992 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:39.099099 (+ 2107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:39.099103 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:39.099106 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:36:39.099137 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:39.101013 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:39.101551 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:39.101555 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:39.101557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:39.101651 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:39.102072 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:39.102076 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:39.102078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:39.102335 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:39.102448 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:39.102835 (+ 387us) server_negotiation.cc:299] Negotiation successful
0619 14:36:39.102962 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 14:36:39.105564 24507 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7074277d8a324bb594b81b92043bb66a [term 1 FOLLOWER]: Refusing update from remote peer 0a10accd949c45869b4e4fd49f0ea5fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:36:39.106238 24543 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0a10accd949c45869b4e4fd49f0ea5fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7074277d8a324bb594b81b92043bb66a" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 45105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:36:39.110706 24580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7074277d8a324bb594b81b92043bb66a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a10accd949c45869b4e4fd49f0ea5fc. Latest consensus state: current_term: 1 leader_uuid: "0a10accd949c45869b4e4fd49f0ea5fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a10accd949c45869b4e4fd49f0ea5fc" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41993 } } peers { permanent_uuid: "958e5a1371894edda1b2142bbb85a482" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 43219 } } peers { permanent_uuid: "7074277d8a324bb594b81b92043bb66a" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 45105 } } }
I20260619 14:36:39.110831 24580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7074277d8a324bb594b81b92043bb66a [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:36:39.111878 24431 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 958e5a1371894edda1b2142bbb85a482 [term 1 FOLLOWER]: Refusing update from remote peer 0a10accd949c45869b4e4fd49f0ea5fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:36:39.112495 24543 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0a10accd949c45869b4e4fd49f0ea5fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "958e5a1371894edda1b2142bbb85a482" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 43219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 14:36:39.119927 24591 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7074277d8a324bb594b81b92043bb66a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:36:39.120114 24591 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:36:39.118048 24575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a10accd949c45869b4e4fd49f0ea5fc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0a10accd949c45869b4e4fd49f0ea5fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a10accd949c45869b4e4fd49f0ea5fc" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41993 } } peers { permanent_uuid: "958e5a1371894edda1b2142bbb85a482" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 43219 } } peers { permanent_uuid: "7074277d8a324bb594b81b92043bb66a" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 45105 } } }
I20260619 14:36:39.120431 24575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a10accd949c45869b4e4fd49f0ea5fc [sys.catalog]: This master's current role is: LEADER
I20260619 14:36:39.121260 24542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 958e5a1371894edda1b2142bbb85a482 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a10accd949c45869b4e4fd49f0ea5fc. Latest consensus state: current_term: 1 leader_uuid: "0a10accd949c45869b4e4fd49f0ea5fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a10accd949c45869b4e4fd49f0ea5fc" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41993 } } peers { permanent_uuid: "958e5a1371894edda1b2142bbb85a482" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 43219 } } peers { permanent_uuid: "7074277d8a324bb594b81b92043bb66a" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 45105 } } }
I20260619 14:36:39.121392 24542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 958e5a1371894edda1b2142bbb85a482 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:36:39.122356 24592 mvcc.cc:205] Tried to move back new op lower bound from 7298579657098719232 to 7298579657012236288. Current Snapshot: MvccSnapshot[applied={T|T < 7298579657098719232}]
I20260619 14:36:39.122722 24578 mvcc.cc:205] Tried to move back new op lower bound from 7298579657098719232 to 7298579657012236288. Current Snapshot: MvccSnapshot[applied={T|T < 7298579657098719232}]
I20260619 14:36:39.122893 24580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7074277d8a324bb594b81b92043bb66a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0a10accd949c45869b4e4fd49f0ea5fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a10accd949c45869b4e4fd49f0ea5fc" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41993 } } peers { permanent_uuid: "958e5a1371894edda1b2142bbb85a482" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 43219 } } peers { permanent_uuid: "7074277d8a324bb594b81b92043bb66a" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 45105 } } }
I20260619 14:36:39.123008 24580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7074277d8a324bb594b81b92043bb66a [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:36:39.123476 24543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a10accd949c45869b4e4fd49f0ea5fc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0a10accd949c45869b4e4fd49f0ea5fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a10accd949c45869b4e4fd49f0ea5fc" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41993 } } peers { permanent_uuid: "958e5a1371894edda1b2142bbb85a482" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 43219 } } peers { permanent_uuid: "7074277d8a324bb594b81b92043bb66a" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 45105 } } }
I20260619 14:36:39.123601 24543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a10accd949c45869b4e4fd49f0ea5fc [sys.catalog]: This master's current role is: LEADER
I20260619 14:36:39.124531 24542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 958e5a1371894edda1b2142bbb85a482 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0a10accd949c45869b4e4fd49f0ea5fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a10accd949c45869b4e4fd49f0ea5fc" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 41993 } } peers { permanent_uuid: "958e5a1371894edda1b2142bbb85a482" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 43219 } } peers { permanent_uuid: "7074277d8a324bb594b81b92043bb66a" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 45105 } } }
I20260619 14:36:39.124648 24542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 958e5a1371894edda1b2142bbb85a482 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:36:39.130084 24577 catalog_manager.cc:1347] Generated new cluster ID: 3ec554b2a34d4d4f9b8998a8a30c4f36
I20260619 14:36:39.130198 24577 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:36:39.133723 24594 mvcc.cc:205] Tried to move back new op lower bound from 7298579657098719232 to 7298579657012236288. Current Snapshot: MvccSnapshot[applied={T|T < 7298579657098719232}]
I20260619 14:36:39.176285 24577 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:36:39.177659 24577 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:36:39.193369 24577 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a10accd949c45869b4e4fd49f0ea5fc: Generated new TSK 0
I20260619 14:36:39.194365 24577 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 14:36:39.229773 24385 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:36:39.226681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53788 (local address 127.2.2.62:41993)
0619 14:36:39.226815 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:39.226820 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:39.228646 (+ 1826us) server_negotiation.cc:407] Connection header received
0619 14:36:39.228872 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:39.228875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:39.228929 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:39.228996 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:39.229635 (+ 639us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53788: BlockingRecv error: recv got EOF from 127.0.0.1:53788 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260619 14:36:39.231870 24537 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:36:39.227428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54672 (local address 127.2.2.60:45105)
0619 14:36:39.227545 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:39.227549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:39.230220 (+ 2671us) server_negotiation.cc:407] Connection header received
0619 14:36:39.230837 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:39.230841 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:39.230894 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:39.230949 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:39.231742 (+ 793us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54672: BlockingRecv error: recv got EOF from 127.0.0.1:54672 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260619 14:36:39.232628 24462 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:36:39.228022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36270 (local address 127.2.2.61:43219)
0619 14:36:39.228136 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:39.228140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:39.230419 (+ 2279us) server_negotiation.cc:407] Connection header received
0619 14:36:39.231099 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:39.231103 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:39.231152 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:39.231201 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:39.232509 (+ 1308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36270: BlockingRecv error: recv got EOF from 127.0.0.1:36270 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260619 14:36:39.237412 24462 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:36:39.235917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36282 (local address 127.2.2.61:43219)
0619 14:36:39.236034 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:39.236038 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:39.236481 (+ 443us) server_negotiation.cc:407] Connection header received
0619 14:36:39.236696 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:39.236700 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:39.236769 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:39.236824 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:39.237310 (+ 486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36282: BlockingRecv error: recv got EOF from 127.0.0.1:36282 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260619 14:36:39.242623 24462 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:36:39.241050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36298 (local address 127.2.2.61:43219)
0619 14:36:39.241154 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:39.241158 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:39.241703 (+ 545us) server_negotiation.cc:407] Connection header received
0619 14:36:39.241885 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:39.241889 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:39.241939 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:39.242004 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:39.242513 (+ 509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36298: BlockingRecv error: recv got EOF from 127.0.0.1:36298 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260619 14:36:39.248440 24462 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:36:39.246901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36302 (local address 127.2.2.61:43219)
0619 14:36:39.247017 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:39.247021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:39.247533 (+ 512us) server_negotiation.cc:407] Connection header received
0619 14:36:39.247725 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:39.247729 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:39.247776 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:39.247826 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:39.248332 (+ 506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36302: BlockingRecv error: recv got EOF from 127.0.0.1:36302 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260619 14:36:39.256705 24462 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:36:39.255052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36314 (local address 127.2.2.61:43219)
0619 14:36:39.255180 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:39.255184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:39.255730 (+ 546us) server_negotiation.cc:407] Connection header received
0619 14:36:39.255930 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:39.255934 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:39.256001 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:39.256055 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:39.256595 (+ 540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36314: BlockingRecv error: recv got EOF from 127.0.0.1:36314 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260619 14:36:39.269215 24462 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:36:39.267462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36328 (local address 127.2.2.61:43219)
0619 14:36:39.267587 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:39.267593 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:39.268165 (+ 572us) server_negotiation.cc:407] Connection header received
0619 14:36:39.268382 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:39.268386 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:39.268444 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:39.268498 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:39.269095 (+ 597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36328: BlockingRecv error: recv got EOF from 127.0.0.1:36328 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260619 14:36:39.289666 24462 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:36:39.288249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36344 (local address 127.2.2.61:43219)
0619 14:36:39.288370 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:39.288375 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:39.288616 (+ 241us) server_negotiation.cc:407] Connection header received
0619 14:36:39.288757 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:39.288760 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:39.288815 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:39.288908 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:39.289537 (+ 629us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36344: BlockingRecv error: recv got EOF from 127.0.0.1:36344 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260619 14:36:39.324816 24462 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:36:39.323749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36352 (local address 127.2.2.61:43219)
0619 14:36:39.323913 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:39.323917 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:39.324045 (+ 128us) server_negotiation.cc:407] Connection header received
0619 14:36:39.324154 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:39.324158 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:39.324211 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:39.324309 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:39.324716 (+ 407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36352: BlockingRecv error: recv got EOF from 127.0.0.1:36352 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260619 14:36:39.392457 24462 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:36:39.391201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36364 (local address 127.2.2.61:43219)
0619 14:36:39.391386 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:39.391391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:39.391532 (+ 141us) server_negotiation.cc:407] Connection header received
0619 14:36:39.391714 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:39.391717 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:39.391774 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:39.391867 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:39.392338 (+ 471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36364: BlockingRecv error: recv got EOF from 127.0.0.1:36364 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260619 14:36:39.523661 24462 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:36:39.522556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36378 (local address 127.2.2.61:43219)
0619 14:36:39.522703 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:39.522707 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:39.522840 (+ 133us) server_negotiation.cc:407] Connection header received
0619 14:36:39.522960 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:39.522962 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:39.523011 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:39.523094 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:39.523554 (+ 460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36378: BlockingRecv error: recv got EOF from 127.0.0.1:36378 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260619 14:36:39.783109 24462 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:36:39.782101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36384 (local address 127.2.2.61:43219)
0619 14:36:39.782293 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:39.782296 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:39.782314 (+ 18us) server_negotiation.cc:407] Connection header received
0619 14:36:39.782452 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:39.782455 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:39.782503 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:39.782587 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:39.783013 (+ 426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36384: BlockingRecv error: recv got EOF from 127.0.0.1:36384 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
I20260619 14:36:40.059790 24554 catalog_manager.cc:1259] Loaded cluster ID: 3ec554b2a34d4d4f9b8998a8a30c4f36
I20260619 14:36:40.059878 24554 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 958e5a1371894edda1b2142bbb85a482: loading cluster ID for follower catalog manager: success
I20260619 14:36:40.061504 24554 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 958e5a1371894edda1b2142bbb85a482: acquiring CA information for follower catalog manager: success
I20260619 14:36:40.062674 24554 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 958e5a1371894edda1b2142bbb85a482: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 14:36:40.121449 24591 catalog_manager.cc:1259] Loaded cluster ID: 3ec554b2a34d4d4f9b8998a8a30c4f36
I20260619 14:36:40.121541 24591 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7074277d8a324bb594b81b92043bb66a: loading cluster ID for follower catalog manager: success
I20260619 14:36:40.123301 24591 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7074277d8a324bb594b81b92043bb66a: acquiring CA information for follower catalog manager: success
I20260619 14:36:40.124557 24591 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7074277d8a324bb594b81b92043bb66a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 14:36:40.301492 24665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:40.297221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36396 (local address 127.2.2.61:43219)
0619 14:36:40.297452 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:40.297456 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:40.297471 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:36:40.297576 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:40.297579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:40.297756 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:40.297857 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:36:40.298384 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:40.298977 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:40.299941 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:40.300159 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:40.300237 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:40.300612 (+ 375us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:36:40.301045 (+ 433us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:36:40.301134 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:40.301242 (+ 108us) server_negotiation.cc:299] Negotiation successful
0619 14:36:40.301294 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":69,"threads_started":1}
I20260619 14:36:40.308851 24671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:40.305225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54684 (local address 127.2.2.60:45105)
0619 14:36:40.305471 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:40.305474 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:40.305531 (+ 57us) server_negotiation.cc:407] Connection header received
0619 14:36:40.305728 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:40.305731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:40.305852 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:40.305962 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:36:40.306257 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:40.306772 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:40.307565 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:40.307775 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:40.307851 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:40.308104 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:36:40.308528 (+ 424us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:36:40.308620 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:40.308671 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 14:36:40.308729 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":109,"threads_started":1}
I20260619 14:36:40.315768 24665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:40.311772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36410 (local address 127.2.2.61:43219)
0619 14:36:40.311962 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:40.311969 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:40.312374 (+ 405us) server_negotiation.cc:407] Connection header received
0619 14:36:40.312512 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:40.312514 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:40.312576 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:40.312666 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:36:40.312971 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:40.313526 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:40.314546 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:40.314863 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:40.314963 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:40.315317 (+ 354us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:36:40.315407 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:36:40.315476 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:40.315561 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 14:36:40.315618 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 14:36:40.315948 24671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:40.312129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54700 (local address 127.2.2.60:45105)
0619 14:36:40.312268 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:40.312273 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:40.312478 (+ 205us) server_negotiation.cc:407] Connection header received
0619 14:36:40.312653 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:40.312658 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:40.312706 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:40.312771 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:36:40.313117 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:40.313635 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:40.314816 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:40.315123 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:40.315217 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:40.315472 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:36:40.315555 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:36:40.315621 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:40.315769 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 14:36:40.315822 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 14:36:40.317625 24678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:40.311709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53796 (local address 127.2.2.62:41993)
0619 14:36:40.312111 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:40.312115 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:40.312129 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:36:40.312239 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:40.312247 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:40.312432 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:40.312551 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:36:40.313652 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:40.314503 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:40.315619 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:40.316161 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:40.316757 (+ 596us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:40.316911 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:36:40.317213 (+ 302us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:36:40.317284 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:40.317387 (+ 103us) server_negotiation.cc:299] Negotiation successful
0619 14:36:40.317459 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":75,"threads_started":1}
I20260619 14:36:40.324810 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 24314
I20260619 14:36:40.331308 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 24389
I20260619 14:36:40.337590 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 24464
2026-06-19T14:36:40Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3134 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:36:40 dist-test-slave-70rm krb5kdc[24682](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:36:40 dist-test-slave-70rm krb5kdc[24682](info): set up 2 sockets
Jun 19 14:36:40 dist-test-slave-70rm krb5kdc[24682](info): commencing operation
krb5kdc: starting...
W20260619 14:36:41.382627 2056 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/krb5kdc/test-user.keytab.
Jun 19 14:36:41 dist-test-slave-70rm krb5kdc[24682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879801, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:36:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:36:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.2.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab.
Entry for principal kudu/127.2.2.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab.
I20260619 14:36:41.505889 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:46187
--webserver_interface=127.2.2.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:36861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.2.62:46187,127.2.2.61:39375,127.2.2.60:43175
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:36:41.618345 24695 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 14:36:41.618654 24695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:36:41.618677 24695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:36:41.618767 24695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:36:41.622849 24695 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:36:41.622912 24695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:36:41.622932 24695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:36:41.622948 24695 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:36:41.622968 24695 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:36:41.628391 24695 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:36861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.2.62:46187,127.2.2.61:39375,127.2.2.60:43175
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.62.keytab
--principal=kudu/127.2.2.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.62:46187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.2.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:36:41.629527 24695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:36:41.630604 24695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:36:41.639078 24701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:41.639057 24700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:41.639081 24703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:41.639309 24695 server_base.cc:1034] running on GCE node
I20260619 14:36:41.640046 24695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:36:41.641196 24695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:36:41.642381 24695 hybrid_clock.cc:648] HybridClock initialized: now 1781879801642356 us; error 52 us; skew 500 ppm
Jun 19 14:36:41 dist-test-slave-70rm krb5kdc[24682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879801, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:36:41.645131 24695 init.cc:377] Logged in from keytab as kudu/127.2.2.62@KRBTEST.COM (short username kudu)
I20260619 14:36:41.645946 24695 webserver.cc:460] Webserver started at http://127.2.2.62:37903/ using document root <none> and password file <none>
I20260619 14:36:41.646518 24695 fs_manager.cc:359] Metadata directory not provided
I20260619 14:36:41.646581 24695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:36:41.646740 24695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:36:41.648645 24695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/minicluster-data/master-0/data/instance:
uuid: "0a53105cc0c1496090bb9d5e811eff6e"
format_stamp: "Formatted at 2026-06-19 14:36:41 on dist-test-slave-70rm"
server_key: "b54b97b2907d1d97a3622962ee7d47f9"
server_key_iv: "c9b753953eb3f5d5425fae8fc9ba3ab4"
server_key_version: "encryptionkey@0"
I20260619 14:36:41.649154 24695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/minicluster-data/master-0/wal/instance:
uuid: "0a53105cc0c1496090bb9d5e811eff6e"
format_stamp: "Formatted at 2026-06-19 14:36:41 on dist-test-slave-70rm"
server_key: "b54b97b2907d1d97a3622962ee7d47f9"
server_key_iv: "c9b753953eb3f5d5425fae8fc9ba3ab4"
server_key_version: "encryptionkey@0"
I20260619 14:36:41.652747 24695 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 14:36:41.655100 24710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:41.656392 24695 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:36:41.656494 24695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/minicluster-data/master-0/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/minicluster-data/master-0/wal
uuid: "0a53105cc0c1496090bb9d5e811eff6e"
format_stamp: "Formatted at 2026-06-19 14:36:41 on dist-test-slave-70rm"
server_key: "b54b97b2907d1d97a3622962ee7d47f9"
server_key_iv: "c9b753953eb3f5d5425fae8fc9ba3ab4"
server_key_version: "encryptionkey@0"
I20260619 14:36:41.656579 24695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:36:41.696501 24695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:36:41.699862 24695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:36:41.700030 24695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:36:41.707381 24695 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.62:46187
I20260619 14:36:41.707705 24762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.62:46187 every 8 connection(s)
I20260619 14:36:41.708370 24695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/minicluster-data/master-0/data/info.pb
I20260619 14:36:41.711205 24763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:36:41.713008 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 24695
I20260619 14:36:41.713163 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/minicluster-data/master-0/wal/instance
I20260619 14:36:41.713469 2056 external_mini_cluster.cc:1428] Setting key 9f61bd98ba5737bd89480348c4576dd3
I20260619 14:36:41.715346 24763 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.2.62" port: 46187 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.2.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.61@KRBTEST.COM" created.
Jun 19 14:36:41 dist-test-slave-70rm krb5kdc[24682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879801, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.62@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:36:41.733177 24767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:41.719493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60218 (local address 127.2.2.62:46187)
0619 14:36:41.719771 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:41.719780 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:41.719815 (+ 35us) server_negotiation.cc:407] Connection header received
0619 14:36:41.721220 (+ 1405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:41.721225 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:41.721465 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:41.721625 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:41.722733 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:41.723812 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:41.724900 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:41.725349 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:41.725599 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:41.728021 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:41.728027 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:41.728038 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:36:41.728061 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:41.729844 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:41.730446 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:41.730452 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:41.730453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:41.730572 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:41.730893 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:41.730898 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:41.730900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:41.731287 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:41.731433 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:41.731804 (+ 371us) server_negotiation.cc:299] Negotiation successful
0619 14:36:41.732063 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":173,"thread_start_us":106,"threads_started":1}
I20260619 14:36:41.733177 24766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:41.718414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46187 (local address 127.0.0.1:60218)
0619 14:36:41.719554 (+ 1140us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:41.719630 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0619 14:36:41.720860 (+ 1230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:36:41.721692 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:36:41.721698 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:36:41.722211 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:36:41.722555 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:41.722571 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:41.723973 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:41.723976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:41.724743 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:41.724781 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:41.725457 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:41.725512 (+ 55us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:41.725651 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:41.726332 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:36:41.726349 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:36:41.727851 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:36:41.729977 (+ 2126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:36:41.729988 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:36:41.729998 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:36:41.730322 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:36:41.730670 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:36:41.730675 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:36:41.730677 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:36:41.730799 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:36:41.731403 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:36:41.731410 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:36:41.731564 (+ 154us) client_negotiation.cc:769] Sending connection context
0619 14:36:41.731732 (+ 168us) client_negotiation.cc:240] Negotiation successful
0619 14:36:41.732123 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":872,"mutex_wait_us":113,"spinlock_wait_cycles":6912,"thread_start_us":171,"threads_started":1}
I20260619 14:36:41.736104 24763 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.2.61" port: 39375 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:36:41.736888 24766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:41.736617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.61:39375 (local address 127.0.0.1:46210)
0619 14:36:41.736750 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:41.736821 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.61:39375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 14:36:41.737115 24711 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.2.61:39375: connect: Connection refused (error 111)
I20260619 14:36:41.738030 24766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:41.737778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.61:39375 (local address 127.0.0.1:46220)
0619 14:36:41.737882 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:41.737934 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.61:39375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 14:36:41.738214 24763 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.2.61:39375: Network error: Client connection negotiation failed: client connection to 127.2.2.61:39375: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.61.keytab.
Entry for principal kudu/127.2.2.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.61.keytab.
I20260619 14:36:41.741571 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.61:39375
--webserver_interface=127.2.2.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.61.keytab
--principal=kudu/127.2.2.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:36861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.2.62:46187,127.2.2.61:39375,127.2.2.60:43175
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:36:41.787525 24763 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.2.61" port: 39375 } attempt: 1
I20260619 14:36:41.788436 24766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:41.788136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.61:39375 (local address 127.0.0.1:46230)
0619 14:36:41.788290 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:41.788360 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.61:39375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260619 14:36:41.789214 24766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:41.788994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.61:39375 (local address 127.0.0.1:46242)
0619 14:36:41.789103 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:41.789159 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.61:39375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 14:36:41.789372 24763 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.2.61:39375: Network error: Client connection negotiation failed: client connection to 127.2.2.61:39375: connect: Connection refused (error 111)
W20260619 14:36:41.852989 24770 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 14:36:41.853271 24770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:36:41.853335 24770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:36:41.853425 24770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:36:41.857416 24770 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:36:41.857471 24770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:36:41.857491 24770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:36:41.857507 24770 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:36:41.857522 24770 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:36:41.857532 24763 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.2.61" port: 39375 } attempt: 2
I20260619 14:36:41.858408 24766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:41.858099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.61:39375 (local address 127.0.0.1:46258)
0619 14:36:41.858260 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:41.858331 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.61:39375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 14:36:41.859179 24766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:41.858943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.61:39375 (local address 127.0.0.1:46274)
0619 14:36:41.859071 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:41.859125 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.61:39375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 14:36:41.859380 24763 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.2.61:39375: Network error: Client connection negotiation failed: client connection to 127.2.2.61:39375: connect: Connection refused (error 111)
I20260619 14:36:41.863261 24770 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:36861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.2.62:46187,127.2.2.61:39375,127.2.2.60:43175
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.61.keytab
--principal=kudu/127.2.2.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.61:39375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.2.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:36:41.864308 24770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:36:41.865360 24770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:36:41.874259 24778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:41.874500 24776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:41.874385 24775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:41.874382 24770 server_base.cc:1034] running on GCE node
I20260619 14:36:41.875130 24770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:36:41.876333 24770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:36:41.877519 24770 hybrid_clock.cc:648] HybridClock initialized: now 1781879801877493 us; error 44 us; skew 500 ppm
Jun 19 14:36:41 dist-test-slave-70rm krb5kdc[24682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879801, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:36:41.880347 24770 init.cc:377] Logged in from keytab as kudu/127.2.2.61@KRBTEST.COM (short username kudu)
I20260619 14:36:41.881251 24770 webserver.cc:460] Webserver started at http://127.2.2.61:37569/ using document root <none> and password file <none>
I20260619 14:36:41.881822 24770 fs_manager.cc:359] Metadata directory not provided
I20260619 14:36:41.881912 24770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:36:41.882200 24770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:36:41.884258 24770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/minicluster-data/master-1/data/instance:
uuid: "15aaef5aa5b148399897a7707b3a0599"
format_stamp: "Formatted at 2026-06-19 14:36:41 on dist-test-slave-70rm"
server_key: "8f58370e35efe617f9a0569d80f05a83"
server_key_iv: "d3f7ea286d90dc64aa5fae28707d8eb9"
server_key_version: "encryptionkey@0"
I20260619 14:36:41.884832 24770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/minicluster-data/master-1/wal/instance:
uuid: "15aaef5aa5b148399897a7707b3a0599"
format_stamp: "Formatted at 2026-06-19 14:36:41 on dist-test-slave-70rm"
server_key: "8f58370e35efe617f9a0569d80f05a83"
server_key_iv: "d3f7ea286d90dc64aa5fae28707d8eb9"
server_key_version: "encryptionkey@0"
I20260619 14:36:41.888656 24770 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260619 14:36:41.891227 24785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:41.892570 24770 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:36:41.892726 24770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/minicluster-data/master-1/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/minicluster-data/master-1/wal
uuid: "15aaef5aa5b148399897a7707b3a0599"
format_stamp: "Formatted at 2026-06-19 14:36:41 on dist-test-slave-70rm"
server_key: "8f58370e35efe617f9a0569d80f05a83"
server_key_iv: "d3f7ea286d90dc64aa5fae28707d8eb9"
server_key_version: "encryptionkey@0"
I20260619 14:36:41.892853 24770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:36:41.916659 24770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:36:41.920224 24770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:36:41.920451 24770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:36:41.927645 24770 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.61:39375
I20260619 14:36:41.927928 24837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.61:39375 every 8 connection(s)
I20260619 14:36:41.928740 24770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/minicluster-data/master-1/data/info.pb
I20260619 14:36:41.931629 24838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:36:41.935490 24838 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.2.62" port: 46187 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:36:41.938197 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 24770
I20260619 14:36:41.938357 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/minicluster-data/master-1/wal/instance
I20260619 14:36:41.938692 2056 external_mini_cluster.cc:1428] Setting key a5721d241fc5cc3dd38a7cb7aada70a9
Jun 19 14:36:41 dist-test-slave-70rm krb5kdc[24682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879801, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.61@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.2.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.2.60@KRBTEST.COM" created.
I20260619 14:36:41.950328 24767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:41.937387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60224 (local address 127.2.2.62:46187)
0619 14:36:41.937523 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:41.937526 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:41.938463 (+ 937us) server_negotiation.cc:407] Connection header received
0619 14:36:41.939957 (+ 1494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:41.939963 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:41.940035 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:41.940131 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:41.941432 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:41.941958 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:41.943222 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:41.943462 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:41.943575 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:41.946211 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:41.946214 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:41.946216 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:36:41.946240 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:41.947827 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:41.948420 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:41.948424 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:41.948426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:41.948544 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:41.948891 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:41.948895 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:41.948897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:41.949103 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:41.949261 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:41.949882 (+ 621us) server_negotiation.cc:299] Negotiation successful
0619 14:36:41.950032 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 14:36:41.951056 24840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:41.937681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46187 (local address 127.0.0.1:60224)
0619 14:36:41.938277 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:41.938346 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0619 14:36:41.939688 (+ 1342us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:36:41.940338 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:36:41.940349 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:36:41.940811 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:36:41.941279 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:41.941291 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:41.942093 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:41.942097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:41.943102 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:41.943111 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:41.943559 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:41.943615 (+ 56us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:41.943779 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:41.944362 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:36:41.944378 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:36:41.946021 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:36:41.947970 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:36:41.947982 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:36:41.947998 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:36:41.948313 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:36:41.948636 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:36:41.948639 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:36:41.948641 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:36:41.948797 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:36:41.949278 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:36:41.949287 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:36:41.949622 (+ 335us) client_negotiation.cc:769] Sending connection context
0619 14:36:41.949796 (+ 174us) client_negotiation.cc:240] Negotiation successful
0619 14:36:41.950084 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":158,"threads_started":1}
I20260619 14:36:41.951076 24763 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.2.61" port: 39375 } attempt: 3
I20260619 14:36:41.952732 24838 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.2.61" port: 39375 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:36:41 dist-test-slave-70rm krb5kdc[24682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879801, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.61@KRBTEST.COM for kudu/127.2.2.61@KRBTEST.COM
Jun 19 14:36:41 dist-test-slave-70rm krb5kdc[24682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879801, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.62@KRBTEST.COM for kudu/127.2.2.61@KRBTEST.COM
I20260619 14:36:41.963078 24840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:41.953260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.61:39375 (local address 127.0.0.1:46298)
0619 14:36:41.953432 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:41.953449 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:36:41.953524 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:36:41.953797 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:36:41.953803 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:36:41.953986 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:36:41.954105 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:41.954116 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:41.954872 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:41.954877 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:41.955384 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:41.955391 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:41.956004 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:41.956025 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:41.956137 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:41.956665 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:36:41.956682 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:36:41.958448 (+ 1766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:36:41.961172 (+ 2724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:36:41.961179 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:36:41.961184 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:36:41.961558 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:36:41.961887 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:36:41.961890 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:36:41.961892 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:36:41.962034 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:36:41.962591 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:36:41.962597 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:36:41.962704 (+ 107us) client_negotiation.cc:769] Sending connection context
0619 14:36:41.962790 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 14:36:41.962898 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":1,"spinlock_wait_cycles":10112}
I20260619 14:36:41.963148 24766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:41.951485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.61:39375 (local address 127.0.0.1:46288)
0619 14:36:41.951674 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:41.951691 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:36:41.951766 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:36:41.952518 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:36:41.952524 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:36:41.952717 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:36:41.952869 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:41.952879 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:41.954145 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:41.954149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:41.955685 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:41.955697 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:41.956233 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:41.956249 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:41.956371 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:41.956828 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:36:41.956840 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:36:41.958481 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:36:41.961646 (+ 3165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:36:41.961651 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:36:41.961653 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:36:41.961896 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:36:41.962193 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:36:41.962200 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:36:41.962202 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:36:41.962275 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:36:41.962642 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:36:41.962646 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:36:41.962734 (+ 88us) client_negotiation.cc:769] Sending connection context
0619 14:36:41.962829 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 14:36:41.962977 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110}
I20260619 14:36:41.963246 24844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:41.953320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46298 (local address 127.2.2.61:39375)
0619 14:36:41.953582 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:41.953590 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:41.953602 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:36:41.953635 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:41.953638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:41.953674 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:41.953759 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:41.954249 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:41.954767 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:41.955498 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:41.955897 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:41.955995 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:41.958648 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:41.958654 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:41.958668 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:36:41.958730 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:41.960998 (+ 2268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:41.961687 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:41.961692 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:41.961693 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:41.961764 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:41.962142 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:41.962147 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:41.962150 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:41.962477 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:41.962626 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:41.962898 (+ 272us) server_negotiation.cc:299] Negotiation successful
0619 14:36:41.963035 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"spinlock_wait_cycles":5504,"thread_start_us":77,"threads_started":1}
I20260619 14:36:41.963747 24843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:41.951684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46288 (local address 127.2.2.61:39375)
0619 14:36:41.952118 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:41.952129 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:41.952150 (+ 21us) server_negotiation.cc:407] Connection header received
0619 14:36:41.952191 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:41.952196 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:41.952377 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:41.952509 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:41.952993 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:41.954009 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:41.955806 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:41.956137 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:41.956241 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:41.958640 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:41.958652 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:41.958668 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:36:41.958731 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:41.961546 (+ 2815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:41.961991 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:41.961993 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:41.961994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:41.962064 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:41.962361 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:41.962364 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:41.962366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:41.962529 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:41.963437 (+ 908us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:41.963503 (+ 66us) server_negotiation.cc:299] Negotiation successful
0619 14:36:41.963603 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"spinlock_wait_cycles":7680,"thread_start_us":126,"threads_started":1}
I20260619 14:36:41.965188 24838 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.2.60" port: 43175 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:36:41.965335 24763 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.2.60" port: 43175 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:36:41.965919 24840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:41.965684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.60:43175 (local address 127.0.0.1:33074)
0619 14:36:41.965795 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:41.965867 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.60:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260619 14:36:41.966169 24788 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.2.60:43175: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.2.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.60.keytab.
Entry for principal kudu/127.2.2.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.60.keytab.
I20260619 14:36:41.966759 24766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:41.965785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.60:43175 (local address 127.0.0.1:33086)
0619 14:36:41.965904 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:41.965961 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.60:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 14:36:41.967293 2056 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
/tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.60:43175
--webserver_interface=127.2.2.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.60.keytab
--principal=kudu/127.2.2.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.2.20:36861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.2.62:46187,127.2.2.61:39375,127.2.2.60:43175
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:36:41.967573 24766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:41.967256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.60:43175 (local address 127.0.0.1:33098)
0619 14:36:41.967427 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:41.967491 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.60:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260619 14:36:41.967715 24763 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.2.60:43175: Network error: Client connection negotiation failed: client connection to 127.2.2.60:43175: connect: Connection refused (error 111)
I20260619 14:36:41.968704 24840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:41.966864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.60:43175 (local address 127.0.0.1:33096)
0619 14:36:41.968593 (+ 1729us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:41.968637 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.60:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1665}
W20260619 14:36:41.968919 24838 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.2.60:43175: Network error: Client connection negotiation failed: client connection to 127.2.2.60:43175: connect: Connection refused (error 111)
I20260619 14:36:41.998870 24763 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.2.60" port: 43175 } attempt: 1
I20260619 14:36:41.999823 24766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:41.999466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.60:43175 (local address 127.0.0.1:33110)
0619 14:36:41.999660 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:41.999735 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.60:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260619 14:36:42.000681 24766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:42.000445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.60:43175 (local address 127.0.0.1:33116)
0619 14:36:42.000563 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:42.000624 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.60:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 14:36:42.000818 24763 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.2.60:43175: Network error: Client connection negotiation failed: client connection to 127.2.2.60:43175: connect: Connection refused (error 111)
I20260619 14:36:42.018224 24838 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.2.60" port: 43175 } attempt: 1
I20260619 14:36:42.019138 24840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:42.018821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.60:43175 (local address 127.0.0.1:33132)
0619 14:36:42.018993 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:42.019061 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.60:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260619 14:36:42.019917 24840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:42.019698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.60:43175 (local address 127.0.0.1:33144)
0619 14:36:42.019808 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:42.019861 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.60:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 14:36:42.020046 24838 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.2.60:43175: Network error: Client connection negotiation failed: client connection to 127.2.2.60:43175: connect: Connection refused (error 111)
I20260619 14:36:42.075986 24763 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.2.60" port: 43175 } attempt: 2
I20260619 14:36:42.076916 24766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:42.076580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.60:43175 (local address 127.0.0.1:33152)
0619 14:36:42.076775 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:42.076839 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.60:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260619 14:36:42.077685 24766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:42.077459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.60:43175 (local address 127.0.0.1:33160)
0619 14:36:42.077585 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:42.077629 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.60:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 14:36:42.077802 24763 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.2.60:43175: Network error: Client connection negotiation failed: client connection to 127.2.2.60:43175: connect: Connection refused (error 111)
W20260619 14:36:42.079730 24846 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 14:36:42.080070 24846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:36:42.080130 24846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:36:42.080228 24846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:36:42.084506 24846 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:36:42.084589 24846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:36:42.084657 24846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:36:42.084712 24846 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:36:42.084751 24846 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:36:42.088215 24838 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.2.60" port: 43175 } attempt: 2
I20260619 14:36:42.089123 24840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:42.088796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.60:43175 (local address 127.0.0.1:33166)
0619 14:36:42.088960 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:42.089049 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.60:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260619 14:36:42.089828 24840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:42.089616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.60:43175 (local address 127.0.0.1:33168)
0619 14:36:42.089723 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:42.089775 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.2.60:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 14:36:42.089982 24838 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.2.60:43175: Network error: Client connection negotiation failed: client connection to 127.2.2.60:43175: connect: Connection refused (error 111)
I20260619 14:36:42.090852 24846 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.2.20:36861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.2.62:46187,127.2.2.61:39375,127.2.2.60:43175
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/krb5kdc/kudu_127.2.2.60.keytab
--principal=kudu/127.2.2.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.2.60:43175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.2.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:36:42.091985 24846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:36:42.093061 24846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:36:42.101785 24851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:36:42.101775 24852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:42.101890 24846 server_base.cc:1034] running on GCE node
W20260619 14:36:42.101775 24854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:36:42.102752 24846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:36:42.103916 24846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:36:42.105083 24846 hybrid_clock.cc:648] HybridClock initialized: now 1781879802105065 us; error 32 us; skew 500 ppm
Jun 19 14:36:42 dist-test-slave-70rm krb5kdc[24682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879802, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:36:42.108074 24846 init.cc:377] Logged in from keytab as kudu/127.2.2.60@KRBTEST.COM (short username kudu)
I20260619 14:36:42.109089 24846 webserver.cc:460] Webserver started at http://127.2.2.60:44591/ using document root <none> and password file <none>
I20260619 14:36:42.109670 24846 fs_manager.cc:359] Metadata directory not provided
I20260619 14:36:42.109763 24846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:36:42.110010 24846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:36:42.112157 24846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/minicluster-data/master-2/data/instance:
uuid: "5575ce66f1894871a88c045757788eb1"
format_stamp: "Formatted at 2026-06-19 14:36:42 on dist-test-slave-70rm"
server_key: "9a78d8ff66a544bdbae1beb9afd76a53"
server_key_iv: "bdd8e7787c2335e90c2703ac150dd000"
server_key_version: "encryptionkey@0"
I20260619 14:36:42.112730 24846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/minicluster-data/master-2/wal/instance:
uuid: "5575ce66f1894871a88c045757788eb1"
format_stamp: "Formatted at 2026-06-19 14:36:42 on dist-test-slave-70rm"
server_key: "9a78d8ff66a544bdbae1beb9afd76a53"
server_key_iv: "bdd8e7787c2335e90c2703ac150dd000"
server_key_version: "encryptionkey@0"
I20260619 14:36:42.116611 24846 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:36:42.119237 24861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:36:42.120548 24846 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:36:42.120697 24846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/minicluster-data/master-2/data,/tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/minicluster-data/master-2/wal
uuid: "5575ce66f1894871a88c045757788eb1"
format_stamp: "Formatted at 2026-06-19 14:36:42 on dist-test-slave-70rm"
server_key: "9a78d8ff66a544bdbae1beb9afd76a53"
server_key_iv: "bdd8e7787c2335e90c2703ac150dd000"
server_key_version: "encryptionkey@0"
I20260619 14:36:42.120826 24846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:36:42.144660 24846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:36:42.148018 24846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:36:42.148231 24846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:36:42.155560 24846 rpc_server.cc:307] RPC server started. Bound to: 127.2.2.60:43175
I20260619 14:36:42.155843 24913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.2.60:43175 every 8 connection(s)
I20260619 14:36:42.156656 24846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/minicluster-data/master-2/data/info.pb
I20260619 14:36:42.159693 24914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:36:42.163803 24914 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.2.62" port: 46187 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:36:42.164167 2056 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu as pid 24846
I20260619 14:36:42.164345 2056 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMyx_Zw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879514993456-2056-0/minicluster-data/master-2/wal/instance
I20260619 14:36:42.164619 2056 external_mini_cluster.cc:1428] Setting key b052f2d54c8f6e9790cb949385fd4079
Jun 19 14:36:42 dist-test-slave-70rm krb5kdc[24682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879801, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.2.60@KRBTEST.COM
Jun 19 14:36:42 dist-test-slave-70rm krb5kdc[24682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879802, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.60@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:36:42.176954 24763 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.2.60" port: 43175 } attempt: 3
I20260619 14:36:42.178334 24767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:42.165859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60226 (local address 127.2.2.62:46187)
0619 14:36:42.165997 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:42.166000 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:42.166651 (+ 651us) server_negotiation.cc:407] Connection header received
0619 14:36:42.167545 (+ 894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:42.167548 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:42.167605 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:42.167711 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:42.168673 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:42.169198 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:42.170331 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:42.170616 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:42.170695 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:42.174801 (+ 4106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:42.174805 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:42.174807 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:36:42.174843 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:42.176201 (+ 1358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:42.176690 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:42.176693 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:42.176695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:42.176776 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:42.177202 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:42.177205 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:42.177206 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:42.177403 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:42.177520 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:42.178019 (+ 499us) server_negotiation.cc:299] Negotiation successful
0619 14:36:42.178176 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 14:36:42.179371 24917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:42.165968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.62:46187 (local address 127.0.0.1:60226)
0619 14:36:42.166528 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:42.166568 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 14:36:42.167328 (+ 760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:36:42.167783 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:36:42.167788 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:36:42.168220 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:36:42.168534 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:42.168547 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:42.169305 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:42.169309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:42.170221 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:42.170230 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:42.170706 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:42.170753 (+ 47us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:42.170951 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:42.172790 (+ 1839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:36:42.172807 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:36:42.174653 (+ 1846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:36:42.176318 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:36:42.176328 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:36:42.176339 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:36:42.176581 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:36:42.176876 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:36:42.176879 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:36:42.176881 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:36:42.177025 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:36:42.177579 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:36:42.177587 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:36:42.177761 (+ 174us) client_negotiation.cc:769] Sending connection context
0619 14:36:42.177951 (+ 190us) client_negotiation.cc:240] Negotiation successful
0619 14:36:42.178389 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"mutex_wait_us":65,"thread_start_us":163,"threads_started":1}
I20260619 14:36:42.179371 24918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:42.165968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33176 (local address 127.2.2.60:43175)
0619 14:36:42.166516 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:42.166533 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:42.166570 (+ 37us) server_negotiation.cc:407] Connection header received
0619 14:36:42.167306 (+ 736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:42.167323 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:42.167533 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:42.167714 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:42.168435 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:42.169469 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:42.170453 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:42.170885 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:42.170995 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:42.174254 (+ 3259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:42.174263 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:42.174278 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:36:42.174304 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:42.176240 (+ 1936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:42.176713 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:42.176719 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:42.176720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:42.176824 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:42.177134 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:42.177137 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:42.177139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:42.177661 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:42.177864 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:42.178258 (+ 394us) server_negotiation.cc:299] Negotiation successful
0619 14:36:42.178714 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":312,"server-negotiator.queue_time_us":431,"spinlock_wait_cycles":37120,"thread_start_us":171,"threads_started":1}
I20260619 14:36:42.180967 24914 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.2.61" port: 39375 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:36:42.181313 24838 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.2.60" port: 43175 } attempt: 3
Jun 19 14:36:42 dist-test-slave-70rm krb5kdc[24682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879801, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.62@KRBTEST.COM for kudu/127.2.2.60@KRBTEST.COM
Jun 19 14:36:42 dist-test-slave-70rm krb5kdc[24682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879802, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.60@KRBTEST.COM for kudu/127.2.2.61@KRBTEST.COM
Jun 19 14:36:42 dist-test-slave-70rm krb5kdc[24682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879801, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.61@KRBTEST.COM for kudu/127.2.2.60@KRBTEST.COM
I20260619 14:36:42.188526 24766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:42.177381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.60:43175 (local address 127.0.0.1:33186)
0619 14:36:42.177541 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:42.177557 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:36:42.177623 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:36:42.178351 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:36:42.178354 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:36:42.178536 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:36:42.178635 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:42.178641 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:42.179662 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:42.179664 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:42.180433 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:42.180442 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:42.180977 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:42.180996 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:42.181081 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:42.181954 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:36:42.181969 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:36:42.183434 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:36:42.186647 (+ 3213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:36:42.186650 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:36:42.186652 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:36:42.186917 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:36:42.187251 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:36:42.187256 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:36:42.187259 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:36:42.187346 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:36:42.187879 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:36:42.187886 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:36:42.188001 (+ 115us) client_negotiation.cc:769] Sending connection context
0619 14:36:42.188129 (+ 128us) client_negotiation.cc:240] Negotiation successful
0619 14:36:42.188290 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
Jun 19 14:36:42 dist-test-slave-70rm krb5kdc[24682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879801, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.2.62@KRBTEST.COM
I20260619 14:36:42.189827 24919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:42.177771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33186 (local address 127.2.2.60:43175)
0619 14:36:42.178087 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:42.178100 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:42.178112 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:36:42.178168 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:42.178171 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:42.178213 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:42.178752 (+ 539us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:42.178842 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:42.179573 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:42.180551 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:42.180861 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:42.182652 (+ 1791us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:42.183574 (+ 922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:42.183578 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:42.183580 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:36:42.183601 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:42.186524 (+ 2923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:42.187050 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:42.187053 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:42.187055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:42.187137 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:42.187454 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:42.187460 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:42.187462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:42.187759 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:42.187869 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:42.188781 (+ 912us) server_negotiation.cc:299] Negotiation successful
0619 14:36:42.188927 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":255,"thread_start_us":128,"threads_started":1}
I20260619 14:36:42.192610 24840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:42.181730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.60:43175 (local address 127.0.0.1:33190)
0619 14:36:42.181806 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:42.181819 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:36:42.181903 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:36:42.182397 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:36:42.182400 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:36:42.182599 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:36:42.182708 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:42.182715 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:42.184137 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:42.184143 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:42.185004 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:42.185012 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:42.185856 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:42.185872 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:42.185959 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:42.186656 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:36:42.186670 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:36:42.188338 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:36:42.190618 (+ 2280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:36:42.190621 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:36:42.190623 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:36:42.190974 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:36:42.191537 (+ 563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:36:42.191543 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:36:42.191545 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:36:42.191638 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:36:42.192061 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:36:42.192063 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:36:42.192171 (+ 108us) client_negotiation.cc:769] Sending connection context
0619 14:36:42.192314 (+ 143us) client_negotiation.cc:240] Negotiation successful
0619 14:36:42.192459 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"spinlock_wait_cycles":12160}
I20260619 14:36:42.193009 24844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:42.181562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46300 (local address 127.2.2.61:39375)
0619 14:36:42.181691 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:42.181695 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:42.181895 (+ 200us) server_negotiation.cc:407] Connection header received
0619 14:36:42.182070 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:42.182073 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:42.182149 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:42.182203 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:42.182819 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:42.183661 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:42.184807 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:42.185191 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:42.185292 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:42.189069 (+ 3777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:42.189072 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:42.189075 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:36:42.189097 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:42.191139 (+ 2042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:42.191656 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:42.191662 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:42.191664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:42.191745 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:42.192061 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:42.192067 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:42.192072 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:42.192288 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:42.192362 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:42.192733 (+ 371us) server_negotiation.cc:299] Negotiation successful
0619 14:36:42.192854 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 14:36:42.193109 24918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:42.182051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33190 (local address 127.2.2.60:43175)
0619 14:36:42.182189 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:42.182193 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:42.182205 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:36:42.182235 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:42.182237 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:42.182274 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:42.182818 (+ 544us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:42.182897 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:42.184034 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:42.185117 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:42.185739 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:42.187190 (+ 1451us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:42.188469 (+ 1279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:42.188483 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:42.188486 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:36:42.188509 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:42.190456 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:42.191083 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:42.191086 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:42.191088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:42.191421 (+ 333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:42.191754 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:42.191756 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:42.191758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:42.191938 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:42.192782 (+ 844us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:42.192852 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 14:36:42.192966 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 14:36:42.193465 24917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:42.181497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.61:39375 (local address 127.0.0.1:46300)
0619 14:36:42.181830 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:42.181843 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:36:42.181960 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:36:42.182354 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:36:42.182357 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:36:42.182550 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:36:42.182723 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:42.182733 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:42.183783 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:42.183786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:42.184691 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:42.184700 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:42.185399 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:42.185417 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:42.185822 (+ 405us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:42.186261 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:36:42.186279 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:36:42.188948 (+ 2669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:36:42.191224 (+ 2276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:36:42.191228 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:36:42.191254 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:36:42.191534 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:36:42.191866 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:36:42.191871 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:36:42.191874 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:36:42.191958 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:36:42.192437 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:36:42.192442 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:36:42.192550 (+ 108us) client_negotiation.cc:769] Sending connection context
0619 14:36:42.193181 (+ 631us) client_negotiation.cc:240] Negotiation successful
0619 14:36:42.193313 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":270}
I20260619 14:36:42.193830 24763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a53105cc0c1496090bb9d5e811eff6e: Bootstrap starting.
I20260619 14:36:42.193966 24767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:42.182326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60228 (local address 127.2.2.62:46187)
0619 14:36:42.182434 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:42.182440 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:42.182598 (+ 158us) server_negotiation.cc:407] Connection header received
0619 14:36:42.183711 (+ 1113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:42.183716 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:42.183764 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:42.183819 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:42.184682 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:42.185362 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:42.186554 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:42.186894 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:42.186982 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:42.190024 (+ 3042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:42.190028 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:42.190033 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:36:42.190060 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:42.192037 (+ 1977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:42.192499 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:42.192503 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:42.192504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:42.192582 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:42.192886 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:42.192890 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:42.192891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:42.193109 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:42.193298 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:42.193663 (+ 365us) server_negotiation.cc:299] Negotiation successful
0619 14:36:42.193775 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 14:36:42.194108 24914 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.2.60" port: 43175 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:36:42.196939 24763 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a53105cc0c1496090bb9d5e811eff6e: Neither blocks nor log segments found. Creating new log.
I20260619 14:36:42.197846 24763 log.cc:824] T 00000000000000000000000000000000 P 0a53105cc0c1496090bb9d5e811eff6e: Log is configured to *not* fsync() on all Append() calls
Jun 19 14:36:42 dist-test-slave-70rm krb5kdc[24682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879802, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.2.60@KRBTEST.COM for kudu/127.2.2.60@KRBTEST.COM
I20260619 14:36:42.201768 24838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15aaef5aa5b148399897a7707b3a0599: Bootstrap starting.
I20260619 14:36:42.201854 24763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a53105cc0c1496090bb9d5e811eff6e: No bootstrap required, opened a new log
I20260619 14:36:42.203426 24917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:42.194608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.2.60:43175 (local address 127.0.0.1:33196)
0619 14:36:42.194894 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0619 14:36:42.194908 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:36:42.194996 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:36:42.195305 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:36:42.195308 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:36:42.195481 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:36:42.195561 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:42.195567 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:42.196360 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:42.196363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:42.197166 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:36:42.197178 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:36:42.197666 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:36:42.197685 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:36:42.197773 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:42.198348 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:36:42.198362 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:36:42.199924 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:36:42.201968 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:36:42.201970 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:36:42.201972 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:36:42.202247 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:36:42.202513 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:36:42.202516 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:36:42.202518 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:36:42.202593 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:36:42.202950 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:36:42.202953 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:36:42.203040 (+ 87us) client_negotiation.cc:769] Sending connection context
0619 14:36:42.203133 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 14:36:42.203272 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220}
I20260619 14:36:42.204525 24918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:42.194697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33196 (local address 127.2.2.60:43175)
0619 14:36:42.194806 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:42.194811 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:42.194975 (+ 164us) server_negotiation.cc:407] Connection header received
0619 14:36:42.195080 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:42.195083 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:42.195124 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:42.195206 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:42.195676 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:42.196245 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:42.197291 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:42.197565 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:42.197644 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:42.200080 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:42.200085 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:42.200088 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:36:42.200115 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:42.201851 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:42.202351 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:42.202354 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:42.202356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:42.202427 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:42.202677 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:42.202679 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:42.202681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:42.202859 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:42.203921 (+ 1062us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:42.204270 (+ 349us) server_negotiation.cc:299] Negotiation successful
0619 14:36:42.204388 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 14:36:42.204855 24838 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15aaef5aa5b148399897a7707b3a0599: Neither blocks nor log segments found. Creating new log.
Jun 19 14:36:42 dist-test-slave-70rm krb5kdc[24682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879801, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.2.61@KRBTEST.COM
I20260619 14:36:42.206014 24838 log.cc:824] T 00000000000000000000000000000000 P 15aaef5aa5b148399897a7707b3a0599: Log is configured to *not* fsync() on all Append() calls
I20260619 14:36:42.206493 24763 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a53105cc0c1496090bb9d5e811eff6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a53105cc0c1496090bb9d5e811eff6e" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 46187 } } peers { permanent_uuid: "15aaef5aa5b148399897a7707b3a0599" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 39375 } } peers { permanent_uuid: "5575ce66f1894871a88c045757788eb1" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 43175 } }
I20260619 14:36:42.206840 24763 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a53105cc0c1496090bb9d5e811eff6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a53105cc0c1496090bb9d5e811eff6e, State: Initialized, Role: FOLLOWER
I20260619 14:36:42.207269 24763 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a53105cc0c1496090bb9d5e811eff6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a53105cc0c1496090bb9d5e811eff6e" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 46187 } } peers { permanent_uuid: "15aaef5aa5b148399897a7707b3a0599" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 39375 } } peers { permanent_uuid: "5575ce66f1894871a88c045757788eb1" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 43175 } }
I20260619 14:36:42.208269 24838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15aaef5aa5b148399897a7707b3a0599: No bootstrap required, opened a new log
I20260619 14:36:42.208314 24914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5575ce66f1894871a88c045757788eb1: Bootstrap starting.
I20260619 14:36:42.208907 24763 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a53105cc0c1496090bb9d5e811eff6e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:36:42.210379 24844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:42.198312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46302 (local address 127.2.2.61:39375)
0619 14:36:42.198510 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:42.198515 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:42.198530 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:36:42.198572 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:42.198575 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:42.198623 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:42.198710 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:42.199683 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:42.200379 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:42.203156 (+ 2777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:42.203473 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:42.203553 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:42.206643 (+ 3090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:42.206646 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:42.206649 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:36:42.206676 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:42.208283 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:42.208801 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:42.208804 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:42.208806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:42.208891 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:42.209226 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:42.209229 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:42.209232 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:42.209466 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:42.209641 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:42.210071 (+ 430us) server_negotiation.cc:299] Negotiation successful
0619 14:36:42.210234 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 14:36:42.210944 24914 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5575ce66f1894871a88c045757788eb1: Neither blocks nor log segments found. Creating new log.
I20260619 14:36:42.211784 24914 log.cc:824] T 00000000000000000000000000000000 P 5575ce66f1894871a88c045757788eb1: Log is configured to *not* fsync() on all Append() calls
I20260619 14:36:42.212368 24838 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15aaef5aa5b148399897a7707b3a0599 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a53105cc0c1496090bb9d5e811eff6e" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 46187 } } peers { permanent_uuid: "15aaef5aa5b148399897a7707b3a0599" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 39375 } } peers { permanent_uuid: "5575ce66f1894871a88c045757788eb1" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 43175 } }
I20260619 14:36:42.212601 24838 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15aaef5aa5b148399897a7707b3a0599 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15aaef5aa5b148399897a7707b3a0599, State: Initialized, Role: FOLLOWER
I20260619 14:36:42.212937 24838 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15aaef5aa5b148399897a7707b3a0599 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a53105cc0c1496090bb9d5e811eff6e" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 46187 } } peers { permanent_uuid: "15aaef5aa5b148399897a7707b3a0599" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 39375 } } peers { permanent_uuid: "5575ce66f1894871a88c045757788eb1" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 43175 } }
I20260619 14:36:42.214520 2056 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:36:42.214543 24838 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15aaef5aa5b148399897a7707b3a0599 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 14:36:42.216167 24933 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0a53105cc0c1496090bb9d5e811eff6e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:36:42.216390 24933 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:36:42.214285 24695 master_runner.cc:186] Error getting master registration for 127.2.2.61:39375: OK, instance_id { permanent_uuid: "15aaef5aa5b148399897a7707b3a0599" instance_seqno: 1781879801925675 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 14:36:42.218307 24737 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "0a53105cc0c1496090bb9d5e811eff6e"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60228
I20260619 14:36:42.218595 24737 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a53105cc0c1496090bb9d5e811eff6e [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 14:36:42.218693 24737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a53105cc0c1496090bb9d5e811eff6e [term 0 FOLLOWER]: Advancing to term 1
W20260619 14:36:42.219400 24947 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 15aaef5aa5b148399897a7707b3a0599: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:36:42.219467 24947 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:36:42.220018 24737 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a53105cc0c1496090bb9d5e811eff6e [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a53105cc0c1496090bb9d5e811eff6e" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 46187 } } peers { permanent_uuid: "15aaef5aa5b148399897a7707b3a0599" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 39375 } } peers { permanent_uuid: "5575ce66f1894871a88c045757788eb1" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 43175 } }
I20260619 14:36:42.221033 24737 leader_election.cc:288] T 00000000000000000000000000000000 P 0a53105cc0c1496090bb9d5e811eff6e [CANDIDATE]: Term 1 election: Requested vote from peers 15aaef5aa5b148399897a7707b3a0599 (127.2.2.61:39375), 5575ce66f1894871a88c045757788eb1 (127.2.2.60:43175)
I20260619 14:36:42.222404 24922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a53105cc0c1496090bb9d5e811eff6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a53105cc0c1496090bb9d5e811eff6e" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 46187 } } peers { permanent_uuid: "15aaef5aa5b148399897a7707b3a0599" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 39375 } } peers { permanent_uuid: "5575ce66f1894871a88c045757788eb1" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 43175 } } }
I20260619 14:36:42.222653 24922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a53105cc0c1496090bb9d5e811eff6e [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:36:42.222924 24935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15aaef5aa5b148399897a7707b3a0599 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a53105cc0c1496090bb9d5e811eff6e" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 46187 } } peers { permanent_uuid: "15aaef5aa5b148399897a7707b3a0599" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 39375 } } peers { permanent_uuid: "5575ce66f1894871a88c045757788eb1" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 43175 } } }
I20260619 14:36:42.223097 24935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15aaef5aa5b148399897a7707b3a0599 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:36:42.224711 24812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a53105cc0c1496090bb9d5e811eff6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "15aaef5aa5b148399897a7707b3a0599"
I20260619 14:36:42.224973 24812 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15aaef5aa5b148399897a7707b3a0599 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:36:42.225338 24914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5575ce66f1894871a88c045757788eb1: No bootstrap required, opened a new log
I20260619 14:36:42.226426 24812 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 15aaef5aa5b148399897a7707b3a0599 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a53105cc0c1496090bb9d5e811eff6e in term 1.
I20260619 14:36:42.226869 24711 leader_election.cc:302] T 00000000000000000000000000000000 P 0a53105cc0c1496090bb9d5e811eff6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a53105cc0c1496090bb9d5e811eff6e, 15aaef5aa5b148399897a7707b3a0599; no voters:
I20260619 14:36:42.227180 24922 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a53105cc0c1496090bb9d5e811eff6e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:36:42.227402 24922 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a53105cc0c1496090bb9d5e811eff6e [term 1 LEADER]: Becoming Leader. State: Replica: 0a53105cc0c1496090bb9d5e811eff6e, State: Running, Role: LEADER
I20260619 14:36:42.227738 24922 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a53105cc0c1496090bb9d5e811eff6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a53105cc0c1496090bb9d5e811eff6e" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 46187 } } peers { permanent_uuid: "15aaef5aa5b148399897a7707b3a0599" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 39375 } } peers { permanent_uuid: "5575ce66f1894871a88c045757788eb1" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 43175 } }
I20260619 14:36:42.229739 24922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a53105cc0c1496090bb9d5e811eff6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a53105cc0c1496090bb9d5e811eff6e. Latest consensus state: current_term: 1 leader_uuid: "0a53105cc0c1496090bb9d5e811eff6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a53105cc0c1496090bb9d5e811eff6e" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 46187 } } peers { permanent_uuid: "15aaef5aa5b148399897a7707b3a0599" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 39375 } } peers { permanent_uuid: "5575ce66f1894871a88c045757788eb1" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 43175 } } }
I20260619 14:36:42.229866 24922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a53105cc0c1496090bb9d5e811eff6e [sys.catalog]: This master's current role is: LEADER
I20260619 14:36:42.229862 24914 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5575ce66f1894871a88c045757788eb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a53105cc0c1496090bb9d5e811eff6e" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 46187 } } peers { permanent_uuid: "15aaef5aa5b148399897a7707b3a0599" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 39375 } } peers { permanent_uuid: "5575ce66f1894871a88c045757788eb1" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 43175 } }
I20260619 14:36:42.230355 24914 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5575ce66f1894871a88c045757788eb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5575ce66f1894871a88c045757788eb1, State: Initialized, Role: FOLLOWER
I20260619 14:36:42.227200 24770 master_runner.cc:186] Error getting master registration for 127.2.2.60:43175: OK, instance_id { permanent_uuid: "5575ce66f1894871a88c045757788eb1" instance_seqno: 1781879802153480 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 14:36:42.231237 24770 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 14:36:42.231386 24695 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 14:36:42.231114 24914 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5575ce66f1894871a88c045757788eb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a53105cc0c1496090bb9d5e811eff6e" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 46187 } } peers { permanent_uuid: "15aaef5aa5b148399897a7707b3a0599" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 39375 } } peers { permanent_uuid: "5575ce66f1894871a88c045757788eb1" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 43175 } }
I20260619 14:36:42.231906 24959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5575ce66f1894871a88c045757788eb1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a53105cc0c1496090bb9d5e811eff6e" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 46187 } } peers { permanent_uuid: "15aaef5aa5b148399897a7707b3a0599" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 39375 } } peers { permanent_uuid: "5575ce66f1894871a88c045757788eb1" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 43175 } } }
I20260619 14:36:42.232108 24959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5575ce66f1894871a88c045757788eb1 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:36:42.233474 24888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a53105cc0c1496090bb9d5e811eff6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5575ce66f1894871a88c045757788eb1"
I20260619 14:36:42.233717 24888 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5575ce66f1894871a88c045757788eb1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:36:42.234884 24956 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:36:42.235602 24888 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5575ce66f1894871a88c045757788eb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a53105cc0c1496090bb9d5e811eff6e in term 1.
I20260619 14:36:42.233413 24914 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5575ce66f1894871a88c045757788eb1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:36:42.240095 24956 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:36:42.246019 24844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:42.225300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46310 (local address 127.2.2.61:39375)
0619 14:36:42.225420 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:42.225425 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:42.230420 (+ 4995us) server_negotiation.cc:407] Connection header received
0619 14:36:42.230661 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:42.230667 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:42.230723 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:42.230775 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:42.238238 (+ 7463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:42.238982 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:42.240107 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:42.240409 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:42.240488 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:42.242385 (+ 1897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:42.242389 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:42.242393 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:36:42.242421 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:42.244083 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:42.244580 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:42.244583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:42.244585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:42.244671 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:42.244973 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:42.244978 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:42.244980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:42.245208 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:42.245357 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:42.245750 (+ 393us) server_negotiation.cc:299] Negotiation successful
0619 14:36:42.245871 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260619 14:36:42.248581 24970 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5575ce66f1894871a88c045757788eb1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:36:42.248886 24970 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:36:42.257663 24767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:42.230715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60232 (local address 127.2.2.62:46187)
0619 14:36:42.230815 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:42.230819 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:42.234631 (+ 3812us) server_negotiation.cc:407] Connection header received
0619 14:36:42.235260 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:42.235264 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:42.235315 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:42.235370 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:42.242282 (+ 6912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:42.243036 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:42.246905 (+ 3869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:42.247276 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:42.247349 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:42.250589 (+ 3240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:42.250593 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:42.250596 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:36:42.250626 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:42.252332 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:42.255180 (+ 2848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:42.255185 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:42.255187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:42.255278 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:42.255937 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:42.255940 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:42.255943 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:42.256206 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:42.256336 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:42.257326 (+ 990us) server_negotiation.cc:299] Negotiation successful
0619 14:36:42.257476 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 14:36:42.259840 24918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:42.230377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33206 (local address 127.2.2.60:43175)
0619 14:36:42.230501 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:42.230505 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:42.234420 (+ 3915us) server_negotiation.cc:407] Connection header received
0619 14:36:42.236077 (+ 1657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:42.236080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:42.236132 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:42.236207 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:42.240360 (+ 4153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:42.241128 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:42.246317 (+ 5189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:42.246662 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:42.246756 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:42.252856 (+ 6100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:36:42.252861 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:36:42.252864 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:36:42.252893 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:36:42.254666 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:42.255577 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:42.255581 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:42.255583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:42.255663 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:36:42.256659 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:36:42.256662 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:36:42.256664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:36:42.256873 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:36:42.256993 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:42.259543 (+ 2550us) server_negotiation.cc:299] Negotiation successful
0619 14:36:42.259678 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 14:36:42.264881 24888 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5575ce66f1894871a88c045757788eb1 [term 1 FOLLOWER]: Refusing update from remote peer 0a53105cc0c1496090bb9d5e811eff6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:36:42.265087 24812 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 15aaef5aa5b148399897a7707b3a0599 [term 1 FOLLOWER]: Refusing update from remote peer 0a53105cc0c1496090bb9d5e811eff6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:36:42.265714 24922 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0a53105cc0c1496090bb9d5e811eff6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "15aaef5aa5b148399897a7707b3a0599" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 39375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:36:42.266170 24922 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0a53105cc0c1496090bb9d5e811eff6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5575ce66f1894871a88c045757788eb1" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 43175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:36:42.270711 24935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15aaef5aa5b148399897a7707b3a0599 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a53105cc0c1496090bb9d5e811eff6e. Latest consensus state: current_term: 1 leader_uuid: "0a53105cc0c1496090bb9d5e811eff6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a53105cc0c1496090bb9d5e811eff6e" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 46187 } } peers { permanent_uuid: "15aaef5aa5b148399897a7707b3a0599" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 39375 } } peers { permanent_uuid: "5575ce66f1894871a88c045757788eb1" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 43175 } } }
I20260619 14:36:42.270854 24935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15aaef5aa5b148399897a7707b3a0599 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:36:42.272794 24959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5575ce66f1894871a88c045757788eb1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a53105cc0c1496090bb9d5e811eff6e. Latest consensus state: current_term: 1 leader_uuid: "0a53105cc0c1496090bb9d5e811eff6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a53105cc0c1496090bb9d5e811eff6e" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 46187 } } peers { permanent_uuid: "15aaef5aa5b148399897a7707b3a0599" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 39375 } } peers { permanent_uuid: "5575ce66f1894871a88c045757788eb1" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 43175 } } }
I20260619 14:36:42.272922 24959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5575ce66f1894871a88c045757788eb1 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:36:42.279907 24954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a53105cc0c1496090bb9d5e811eff6e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0a53105cc0c1496090bb9d5e811eff6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a53105cc0c1496090bb9d5e811eff6e" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 46187 } } peers { permanent_uuid: "15aaef5aa5b148399897a7707b3a0599" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 39375 } } peers { permanent_uuid: "5575ce66f1894871a88c045757788eb1" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 43175 } } }
I20260619 14:36:42.280102 24954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a53105cc0c1496090bb9d5e811eff6e [sys.catalog]: This master's current role is: LEADER
I20260619 14:36:42.281694 24971 mvcc.cc:205] Tried to move back new op lower bound from 7298579670068551680 to 7298579669928964096. Current Snapshot: MvccSnapshot[applied={T|T < 7298579670068551680}]
I20260619 14:36:42.282543 24959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5575ce66f1894871a88c045757788eb1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0a53105cc0c1496090bb9d5e811eff6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a53105cc0c1496090bb9d5e811eff6e" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 46187 } } peers { permanent_uuid: "15aaef5aa5b148399897a7707b3a0599" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 39375 } } peers { permanent_uuid: "5575ce66f1894871a88c045757788eb1" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 43175 } } }
I20260619 14:36:42.282660 24959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5575ce66f1894871a88c045757788eb1 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:36:42.286259 24922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a53105cc0c1496090bb9d5e811eff6e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0a53105cc0c1496090bb9d5e811eff6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a53105cc0c1496090bb9d5e811eff6e" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 46187 } } peers { permanent_uuid: "15aaef5aa5b148399897a7707b3a0599" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 39375 } } peers { permanent_uuid: "5575ce66f1894871a88c045757788eb1" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 43175 } } }
I20260619 14:36:42.286396 24922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a53105cc0c1496090bb9d5e811eff6e [sys.catalog]: This master's current role is: LEADER
I20260619 14:36:42.287616 24935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15aaef5aa5b148399897a7707b3a0599 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0a53105cc0c1496090bb9d5e811eff6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a53105cc0c1496090bb9d5e811eff6e" member_type: VOTER last_known_addr { host: "127.2.2.62" port: 46187 } } peers { permanent_uuid: "15aaef5aa5b148399897a7707b3a0599" member_type: VOTER last_known_addr { host: "127.2.2.61" port: 39375 } } peers { permanent_uuid: "5575ce66f1894871a88c045757788eb1" member_type: VOTER last_known_addr { host: "127.2.2.60" port: 43175 } } }
I20260619 14:36:42.287734 24935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15aaef5aa5b148399897a7707b3a0599 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:36:42.287990 24956 catalog_manager.cc:1347] Generated new cluster ID: 8cb4d6f4fe2f4b77af8290e89655b7f5
I20260619 14:36:42.288071 24956 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:36:42.288503 24974 mvcc.cc:205] Tried to move back new op lower bound from 7298579670068551680 to 7298579669928964096. Current Snapshot: MvccSnapshot[applied={T|T < 7298579670068551680}]
I20260619 14:36:42.290333 24973 mvcc.cc:205] Tried to move back new op lower bound from 7298579670068551680 to 7298579669928964096. Current Snapshot: MvccSnapshot[applied={T|T < 7298579670068551680}]
I20260619 14:36:42.320199 24956 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:36:42.321765 24956 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:36:42.336187 24956 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a53105cc0c1496090bb9d5e811eff6e: Generated new TSK 0
I20260619 14:36:42.337335 24956 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 14:36:42.368494 24767 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:36:42.365913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60244 (local address 127.2.2.62:46187)
0619 14:36:42.366048 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:42.366053 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:42.367342 (+ 1289us) server_negotiation.cc:407] Connection header received
0619 14:36:42.367567 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:42.367571 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:42.367626 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:42.367681 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:42.368334 (+ 653us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60244: BlockingRecv error: recv got EOF from 127.0.0.1:60244 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260619 14:36:42.370201 24918 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:36:42.366784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33222 (local address 127.2.2.60:43175)
0619 14:36:42.366895 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:42.366900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:42.369207 (+ 2307us) server_negotiation.cc:407] Connection header received
0619 14:36:42.369383 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:42.369387 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:42.369437 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:42.369501 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:42.370050 (+ 549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33222: BlockingRecv error: recv got EOF from 127.0.0.1:33222 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260619 14:36:42.372303 24844 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:36:42.370836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46316 (local address 127.2.2.61:39375)
0619 14:36:42.370954 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:42.370959 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:42.371409 (+ 450us) server_negotiation.cc:407] Connection header received
0619 14:36:42.371587 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:42.371590 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:42.371638 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:42.371689 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:42.372181 (+ 492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46316: BlockingRecv error: recv got EOF from 127.0.0.1:46316 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260619 14:36:42.383582 24844 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:36:42.378599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46320 (local address 127.2.2.61:39375)
0619 14:36:42.378725 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:42.378732 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:42.382481 (+ 3749us) server_negotiation.cc:407] Connection header received
0619 14:36:42.382725 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:42.382729 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:42.382818 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:42.382874 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:42.383443 (+ 569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46320: BlockingRecv error: recv got EOF from 127.0.0.1:46320 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260619 14:36:42.389353 24844 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:36:42.387654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46322 (local address 127.2.2.61:39375)
0619 14:36:42.387784 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:42.387788 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:42.388318 (+ 530us) server_negotiation.cc:407] Connection header received
0619 14:36:42.388538 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:42.388542 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:42.388599 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:42.388653 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:42.389233 (+ 580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46322: BlockingRecv error: recv got EOF from 127.0.0.1:46322 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260619 14:36:42.395826 24844 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:36:42.394063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46326 (local address 127.2.2.61:39375)
0619 14:36:42.394207 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:42.394211 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:42.394746 (+ 535us) server_negotiation.cc:407] Connection header received
0619 14:36:42.394996 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:42.394999 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:42.395057 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:42.395113 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:42.395706 (+ 593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46326: BlockingRecv error: recv got EOF from 127.0.0.1:46326 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260619 14:36:42.404488 24844 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:36:42.402713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46338 (local address 127.2.2.61:39375)
0619 14:36:42.402871 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:42.402876 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:42.403425 (+ 549us) server_negotiation.cc:407] Connection header received
0619 14:36:42.403674 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:42.403678 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:42.403732 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:42.403788 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:42.404357 (+ 569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46338: BlockingRecv error: recv got EOF from 127.0.0.1:46338 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260619 14:36:42.417157 24844 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:36:42.415327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46354 (local address 127.2.2.61:39375)
0619 14:36:42.415482 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:42.415487 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:42.416055 (+ 568us) server_negotiation.cc:407] Connection header received
0619 14:36:42.416295 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:42.416299 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:42.416361 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:42.416421 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:42.417031 (+ 610us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46354: BlockingRecv error: recv got EOF from 127.0.0.1:46354 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260619 14:36:42.437794 24844 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:36:42.436044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46360 (local address 127.2.2.61:39375)
0619 14:36:42.436185 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:42.436189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:42.436750 (+ 561us) server_negotiation.cc:407] Connection header received
0619 14:36:42.436960 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:42.436965 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:42.437025 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:42.437098 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:42.437676 (+ 578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46360: BlockingRecv error: recv got EOF from 127.0.0.1:46360 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260619 14:36:42.473819 24844 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:36:42.472613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46374 (local address 127.2.2.61:39375)
0619 14:36:42.472762 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:42.472768 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:42.472947 (+ 179us) server_negotiation.cc:407] Connection header received
0619 14:36:42.473052 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:42.473055 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:42.473104 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:42.473180 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:42.473694 (+ 514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46374: BlockingRecv error: recv got EOF from 127.0.0.1:46374 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260619 14:36:42.542431 24844 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:36:42.540325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46384 (local address 127.2.2.61:39375)
0619 14:36:42.540726 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:42.540731 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:42.540854 (+ 123us) server_negotiation.cc:407] Connection header received
0619 14:36:42.541264 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:42.541271 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:42.541343 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:42.541457 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:42.542291 (+ 834us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46384: BlockingRecv error: recv got EOF from 127.0.0.1:46384 (error 108)
Metrics: {"server-negotiator.queue_time_us":297}
W20260619 14:36:42.673688 24844 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:36:42.672433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46386 (local address 127.2.2.61:39375)
0619 14:36:42.672597 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:42.672604 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:42.672805 (+ 201us) server_negotiation.cc:407] Connection header received
0619 14:36:42.673009 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:42.673012 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:42.673061 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:42.673153 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:42.673583 (+ 430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46386: BlockingRecv error: recv got EOF from 127.0.0.1:46386 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260619 14:36:42.932946 24844 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:36:42.931867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46402 (local address 127.2.2.61:39375)
0619 14:36:42.931992 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:42.931995 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:42.932101 (+ 106us) server_negotiation.cc:407] Connection header received
0619 14:36:42.932233 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:42.932236 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:42.932280 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:42.932350 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:36:42.932838 (+ 488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46402: BlockingRecv error: recv got EOF from 127.0.0.1:46402 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 14:36:43.220824 24947 catalog_manager.cc:1259] Loaded cluster ID: 8cb4d6f4fe2f4b77af8290e89655b7f5
I20260619 14:36:43.220929 24947 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 15aaef5aa5b148399897a7707b3a0599: loading cluster ID for follower catalog manager: success
I20260619 14:36:43.222784 24947 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 15aaef5aa5b148399897a7707b3a0599: acquiring CA information for follower catalog manager: success
I20260619 14:36:43.224133 24947 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 15aaef5aa5b148399897a7707b3a0599: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 14:36:43.250233 24970 catalog_manager.cc:1259] Loaded cluster ID: 8cb4d6f4fe2f4b77af8290e89655b7f5
I20260619 14:36:43.250350 24970 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5575ce66f1894871a88c045757788eb1: loading cluster ID for follower catalog manager: success
I20260619 14:36:43.252032 24970 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5575ce66f1894871a88c045757788eb1: acquiring CA information for follower catalog manager: success
I20260619 14:36:43.253170 24970 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5575ce66f1894871a88c045757788eb1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 14:36:43.451577 25044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:43.447204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46418 (local address 127.2.2.61:39375)
0619 14:36:43.447449 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:43.447453 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:43.447480 (+ 27us) server_negotiation.cc:407] Connection header received
0619 14:36:43.447587 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:43.447590 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:43.447712 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:43.447809 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:36:43.448150 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:43.448784 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:43.449855 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:43.450100 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:43.450252 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:43.450633 (+ 381us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:36:43.451077 (+ 444us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:36:43.451167 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:43.451296 (+ 129us) server_negotiation.cc:299] Negotiation successful
0619 14:36:43.451355 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":76,"threads_started":1}
I20260619 14:36:43.458796 25050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:43.454942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33228 (local address 127.2.2.60:43175)
0619 14:36:43.455234 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:43.455238 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:43.455250 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:36:43.455329 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:43.455331 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:43.455446 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:43.455554 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:36:43.455827 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:43.456322 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:43.457217 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:43.457455 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:43.457544 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:43.457802 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:36:43.458326 (+ 524us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:36:43.458424 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:43.458535 (+ 111us) server_negotiation.cc:299] Negotiation successful
0619 14:36:43.458588 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":94,"threads_started":1}
I20260619 14:36:43.466307 25050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:43.462293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33242 (local address 127.2.2.60:43175)
0619 14:36:43.462402 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:43.462414 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:43.462425 (+ 11us) server_negotiation.cc:407] Connection header received
0619 14:36:43.462540 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:43.462543 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:43.462606 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:43.462683 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:36:43.463081 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:43.463797 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:43.465139 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:43.465375 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:43.465491 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:43.465769 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:36:43.465849 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:36:43.465919 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:43.466077 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 14:36:43.466149 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 14:36:43.466562 25044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:43.461951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46434 (local address 127.2.2.61:39375)
0619 14:36:43.462305 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:43.462310 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:43.462327 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:36:43.462391 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:43.462394 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:43.462444 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:43.462520 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:36:43.462781 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:43.463634 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:43.465121 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:43.465451 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:43.465561 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:43.466012 (+ 451us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:36:43.466091 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:36:43.466202 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:43.466349 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 14:36:43.466410 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258}
I20260619 14:36:43.467434 25057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:36:43.462057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60254 (local address 127.2.2.62:46187)
0619 14:36:43.462867 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0619 14:36:43.462872 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:36:43.462886 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:36:43.463043 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:36:43.463047 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:36:43.463191 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:36:43.463317 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:36:43.463571 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:43.464215 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:43.465191 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:36:43.465535 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:36:43.465629 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:36:43.466025 (+ 396us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:36:43.466425 (+ 400us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:36:43.466504 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 14:36:43.467228 (+ 724us) server_negotiation.cc:299] Negotiation successful
0619 14:36:43.467294 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":728,"thread_start_us":86,"threads_started":1}
I20260619 14:36:43.470347 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 24695
I20260619 14:36:43.476832 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 24770
I20260619 14:36:43.482708 2056 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMyx_Zw/build/debug/bin/kudu with pid 24846
2026-06-19T14:36:43Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3144 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6279 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (288482 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST